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FOREWORD 



This Handbook of Adult Curriculim Pevelopaent is based 
lai^gely on presentations prepared for tlie Naticnal Woilcshop on 
tiie Canada Manpower Training Program held in Winnipeg during 
February y 1974. The processes of development and evaluation 
described in the different p^rs i#ere selected because of 
'^ir inmediate practical value and were sometimes Bdapted to 
fit a particular prpgram situation, or the processes grew out 
of the experience of course development and evaluation officers 
woildng at Sa«katdiewan NewStart Incorporated during the years 
1968 to 19 72 and after that at the Training Itesearch aid 
Development Station. Several of the diapters were pil)lished 
early in 1974 in a report entitled Canadian Ot)en Adult 
Learning Systems. For a wider application mid understanding 
ot tne oeveiopment processes described in these papers , the 
pti)lications associated with the different projects referred 
to should be consulted. 



It is hqped that this study will further the discussion 
of alternative instructional designs and delivery systens for 
adult training and should not be construed as a fomal proposal 
by the Department of Manpower and Immigration. 



Hie Training Research aid Develq>ment Station was established 
in Sept^er, 1972 to develq> new methods of oouiselling aid 
training adults. 



V. W. Mullen, 
Acting Director, 
Training Research and 
Novenber, 1974 Development Station 
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aiAFTER I 



CHARACTERISTICS § FORMULATIGN 
OF BHiAVIORAL OBJECTIVES 



A. D. Staith* 



To ensure that the characteristics of Bdiaviora] Objectives and their 
foxnulation be placed in a proper context, it is essoitial to examine the 
four major a^roadies used by most educators when stating their goals. 



APPRQAOIES USED FOR 
EELINEATING EEUCAnCNAL GOALS 

1. Instructor Oriented 

2. Activity Oriented 

3. Learning Oriented 

4. Bdhavior Oriented 
CObservabl - § 
Measurable) 



Far too many educators use idiat may be called the "Instructor Oriented 
^Toadi". Sanple objectives tdiich they develop tend to state "The teacher 
will e]q>lain the causes of the First World War" or "The teadier will lead 
a discussion on the Riel Rd)ellion". In this ai^oadi, the student is 
essentially an observer, usually passive, and the aqdiasis is placed on 
instructor teadiing rather than student Isaming. 

Still other educators structure their curriculun in tenis of a 
sequence of activities in which the studoits will participate. They use, 
idiat has been called, "The Activity Oriented Approach" and develop 
objectives, sudi as "The students will discuss the causes of the First 
World War" or "The students will take a trip to a factory". Educators 
\Aio use this approach tend to feel that their objectives have been 
readied %dien the studaits have participated in all the activities with 
saone degree of involvement. More attention is paid to the structuring 
of the environment for learning rather than the intended outccnies. 

" Manager, Sk'll Development Studies " 
Training Rt. earch and Development Station 

Vaper read at a National Workshop on Individualized Adult Science Education 
in Prince Albert, Saskatchewai] > May, 1972. 
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Another approach lAidi may be used is the 'Teaming Oriented Approach". 
Educators v*io use this approach tend to write objectives, such as 'The 
student will learn the causes of the First World War" or "The student will 
better understand Confederation". I sympathize with educators %rfio use 
this approach because they are honestly attempting to put the enq^iasis 
on the student. However, because they do not use words such as "knowing" - 
and '"understanding'* according to their precise context, their methods are 
often traditional and their efforts tend to the subjective rather than 
the objective. 

The final approach, and one which should be discussed in detail, is 
"The Behavior Oriented ^roach". In this approach, the outcome of 
learning is stated in teims of what the student will be able to do as a 
result of leariiing. In other words, adherents to this apprcadi have 
defined learning as, "An observable change in behavior of the student as 
a result of the learning experience". Those, who believe in this sqjproach, 
define the learning objectives first and then deteimine the learning 
experiences necessary for the student to attain these objectives. 

In sijq>le terms, a behavioral objective is a precise statement of 
yihat a student will be able to do as a result of a unit of instruction. 
There are maany styles of foimulating behavioral objectives. The styles 
are unimportant - vtfiat is i]iqx)rtant is that they contain four essential 
characteristics . 





CHARACTERISTICS 


1. 


Stated from student*s point of view. 


2. 


Stated in observable behavior. 


3. 


Specify conditions under \Aiid\ 




behavior will be perfoimed. 


4. 


Specify acceptable level of per- 




foimance. 



First . Behavioral objectives are expressed in teims of the student 

ratner than that of the teacrier. It must be ensftiasized that behavioral objectives 

are not the teacher's goals - or guides - or aims. They are a statement 

of vAiat the student is to achieve. 

Second , behavioral objectives are always expressed in observable 
behavior. If you cannot see the behavior perfoimed on the part of the 
student, it is not a behavioral objective. 

Third . They contain a statement or statements of the conditions under 
which the behavior will be perfoimed. 

Fourth. A behavioral objective should alw^o^s clearly indicate the 
required standard of perfoimance. 
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Let's take a closer look at each of the four characteristics. 



1. STATED FROM SIUDENT'S 
POINT OF VIBlf 




. . THE SIUDENT WILL 
BE ABLE TO 



Regardless of the style in idiich a behavioral objective is written, 
it should always be fro* the point of view of the student. Teachers 
tiio have been writing instructional aims for years find this particu- 
larly difficult. They have been writing teacher objectives rather than 
student objectives and waaay of then require considerable help In develop- 
ing the skill of writing objectives in teims of the student. 

Another difficulty in writing objectives from a student point of 
view is the difficulty of being precise while still renaining In students' 
terns. 



Ihe first abjective is obviously precise but a student idio cannot 
subtract is not likely to understand it. It is not written from his 
point of view. He would be nore likely to understand the second objective. 




SOLVE FR0BLB4S LIKE THESE: 



578 
-399 



402 
-127 
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2. STATED IN OBSERVABLE BBiAVIOR 

A. FERFOIMANCE TO BE OBSERVED: 

Given a partner and the appropriate 
music, he will perfoim the basic 
fox trot dance steps. 

B« RESULTS OF PERTOSMANCE TO BE OBSERVED: 

Solv& sinqple quadratic equations. 



What is meant by observ^le bdiavior? It can mean one of two things 
or a conbination of both. If you were teaching a student to dance you 
%«ould probably want to observe the actual skill being perfonned. For 
academic work, in massf cases , you would observe the results of the bdiavior 
perfoimance. 

In vocational training, you would probably wish to observe the 
bdiavior and exanine the results frequently. 



BEHAVIORAL WORDS 
und^^tand 

1^ 

appi^^ate 



state 
list 
repair 
construct 
solve 



ejq>lain 
identify 
sketch 
build 
recognize 



Because we have no wiqt of observing a student's brain ^le per- 
foiming, we do not use words like '^understand" or **kn0W**. Instead, 
we use action words idiich dononstrate a student's understanding or 
knowledge. Even many of the words on this list require modification 
to show the actual behavior. For example , you can identify by circling , 
underlining 9 touching , etc. 
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3. CONDITIONS OF BBIAVIOR 

Without References 
Frcm Memory 
Given Tools 
Provided With 
Drawings Of 
With No Warning 
From a Stock Pile 



The third characteristic of behavioral objectives is that they 
specify the conditions under i^ch the bdiavior will be perfoimed. 
Neither the instructor nor the student should have sny illusions about 
the conditions under iHdiich the behavior is to be perfomed. ¥oi* 
exanqple, it irauld be unfair for the student to be told that the 
naterials are to be provided and then e^q)ect hrn to select the most 
appropriate materials from a stoclq>ile. 

Likewise y a student has a right to know whether he has to 
perfom from memory or irfiether he will be allowed the use of his 
reference material. 



4. ACCEPTABLE LEVEL OF PERPOSMANCE 

.. tolerance from -1/8" to 
+1/8" 

901 accuracy 

at least 10 
• • no more than 2 errors 
.. without error 




At some stage » teadiers have to become precise about the required 
standard of perfoimance. All too often this is done by marking the 
examination papers. This is one significant difference in the use of 
bdiavioral objectives. The student is not judged in relation to his 
peers - instead he is judged in relation to a well defined standard. 
Maries or grades may be given to the student for adniinistrative purposes. 
However, most educators believe it is only necessary to substantiate 
•whether or not the student can perfom tlte objective. The belief that 
students relish the competition for high marks or grades is held only 
by instructors and tJie better students ^o are trying for top place. 
The others generally just wish to pass the course as best as they can. 
If you set the level of perfoimance yAiere it is acceptable » then that 
in itself becomes the objective rather than some meaningless percentage 
or grade. 
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Let*s have a look at a behavioral objective and identify the last 
three characteristics. 



(Given a sketch of four levers**) 

the student will be able to 
(classify*) 

(at least 3 of thoa***) 

by inserting the letters A, B 
or C under each diagram 

* Student Behavior 
** Conditions of Behavior 
*** Standards of Bdiavior 



As you see, the student behavior is indicated by an asterisk (*) . 
He will demonstrate his attainment of the objective by inserting - the 
letter A, B or C. The condition of behavior is indicated by **. He is 
provided with a sketdi. 

The standard of perfoimance is shown with tliree asterisks (***) . 
He must have at least three of them correct to achieve this objective* 

Hopefiilly, ny cbjecti .-s regarding the characteristics of be- 
havioral objectives have been accomplished, in the followijig section, 
the foimulation of behavioral objectives will be discussed. • 

There are many styles of writing bdiavioral objectives. Some 
people insist that they must begin with the words, 'The student will 
be able to**. Others believe that this is unnecessary veibiage and 
they insist that they begin with an i]iq;)erative or action veib sudi 
as ''state", "list", "constTuct", etc. Some developers believe that 
all bdiavioral objectives shxnild comnence with the word "given". 
Still others break them down into three distinct parts of conditions, 
actions, and standards. The style is really uninqwrtant; what is 
important is tiiat the teacher aid the students clearly understand the 
intent of the objective. 

An objective is not necessarily good because it has been written 
in bdiavioral teniis. A teacher or instructor i^ho has develc^>ed poor 
or inappropriate test itans will probably develop poor objectives. I 
would like to ^onment on a few of the typical errors i^idli many make 
vtfien they begin to develop behavioral objectives. 
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Probably the most conmon type of error is in the use of numerous low 
level cognitive skills. Pure memory objectives such as list, define, 
state the four principles of, etc., are easy to write. This is under- 
standable because research in the United States regarding actual examin- 
ations has frequently shown a content of 80 to 90 percent of pure knowledge 
questions. *^ . * 

Another frequent ty jr is the creation of objectives which 

are so general or global .... . they are nearly unteachable. Such objectives 
as, the student must be able to communicate',', or "must be able to perform 
mathematics", fall into this category. Objectives of this type need 
further analysis so that they may be broken down into the specific skills 
vrtiich are required to accomplish the general objective. A detailed 
analysis of a course or an occupation will usually reveal a hierarchy of 
objectives. 



IIIFRARQIY 
f AIM OF OOURSc ] 




1 1 


1 n 














1 




ENABLING OBJECTIVES 
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The pro j actual on the previous page shows a typical hierarchy vdiich 
has three levels. The first level shows the aim of the course which is 
probably not written in behavioral terms, but which is in fact, a short 
description of the course . At the next level you would identify teminal 
or global objectives. For an electrician this might be the ability to 
install a 100-amp service entrance. 

At a lower level, multi enabling behavioral objectives can be 
identified. These are generally discrete and quite specific skills. For 
the electrician, these could include skills such as measuring and cutting 
electric cable, making teminal connections, or identifying material. 
The point I wish to make clear is the fact that it is all too easy to 
identify the terminal objectives and then find that they are so broad or 
general that Lhty really are not teachable. 

A third type of error involves the foimulation of all objectives 
so specifically that they are in fact the test item. Objectives sudi as 
'TTie student will be able to list the four characteristics of a behavioral 
objective" are quite specific. It should be noted that this is not a.had 
objective. However, if all the objectives for a course are that specific, 
then there is room for argument that behavioral objectives do not really 
cover the range and depth required of a particular course. 

There are scane administrative short cuts which will lessen the 
difficulties of developing behavioral objectives. The statement, 'TTie 
student will be able to" can be assumed to be part of each objective. 
Similarily, one can say that unless otherwise stated, the student will 
always have his reference books or will always be provided with the tools 
of his trade, etc. 

In conclusion, the following slides will show the value of behavioral 
objectives in teims of the student, the teacher, and the Oirriculum 
developers. 
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Left Projector 



Slide 1 




? 


m 



Slide 2 



1. THEY HELP STUDEOTS 



Slide 3 



1. THEY HELP SHHENTS 



Right Projector 



Slide la 



BEHAVIORAL OBJECTIVES 




Slide 3a 

..KNCM PRECISELY WHAT THEY 
HAVE TO DO 
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Slide 4 



1. •fflEY HELP STUEEWrS 



Slide 4a 

. .KncM precisely what they 

have to do 
. .STUDY MORE EFFECTIVELY 



Slide 5 



1. THEY HELP STODEWTS 



Slide 5a 

. .Iteow precisely what they 

have to do 
. .Stud^ more effectively 
..GQMPRQIEND MORE 

EFFECTIVELY 



Slide 6 



1. THEY HELP STUDENTS 



Slide, 6a 

. .KncM precisely what they 

have to do 
. .Stucty more effectively 
. .Conprehend more 

effectively 
. .TRY OWN SOLUTIONS 
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Slide 


7 


1. 


THEY HELP STUIffiNTS 



Slide 8 



1 . THEY HELP TEAfflERS 



Slide 9 



2. THEY HELP TEACHERS 



Slide 7a 

. .Kmm precisely v*iat th^ 

have to do 
. .Stwty more effectively 
. .ConQ)rehaid more 

effectively 
..Try own solutions 
..KNGW WHEN THEY KNOW 




Slide 9a 

..KNOW PRECISELY WHAT IS TO 
BE TAUGHT 
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Slide 10 



2. THEY HELP TEACHERS 



Slide 10a 

..Know precisely what is 

to be taught 
..ASSIST IN DETEFMINING 

ME1H)D6 



Slide 11 



2. THEY HELP TEACHERS 



Slide 11a 

. .Know precisely what is 

to be taught 
..Assist in deteimining 

methods 
. .HELP IN ASSESSMENT OF 

STUDENT PROGRESS 



Slide 12 



2. THEY HELP TEACHERS 



Slide 12a 

. .Know precisely yihat is 

to be taught 
..Assist in detemining 

methods 
..Help in assessment of 

student progress 
..AID IN SELF EVALUATION 

OF TEACHING TECHNICJUES 
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Slide 13 



2. THEY HELP TEACHERS 



Slide 14 



3. THEY HELP DEVELOPERS 



Slide 15 



3. THEY HELP DEVELOPERS 



Slide 13a 

. .Know precisely what is 
to be taught 

..Assist in deteimining 
methods 

..Help in assessment of 
student progress 

..Aid in self evaluation of 
teaching techniques 

..IF YOU DON'T KNOW WHERE 
YOU ARE GOING YOU ARE 
LIABLE TO END UP SOME- 
PLACE ELSE 




Slide 15a 
. .TO BE PRECISE IN 
CURRICULIM DESIGN 
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Slide le 



3. THEY HELP DEVELOPERS 



Slide 16a 

. .To be precise in 
curriculum design 

. .TO ENSURE THAT TOTAL 
COGNITIVE PROCESS IS 
BEING CONSIDERED IN 
INSTRUCTIONAL PROGRAM 



Slide 17 



3. THEY HELP DEVELOPERS 



Slide 17a 

..To be precise in 
curriculun design 

..To ensure that total 
cognitive process is 
being considered in 
instructional program 

. .TO EVALUATE THE EFFECT- 
IVENESS OF THEIR 
CURRICULUM DESIGN 
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Slide 19 
FAULTS 



Slide 19a 
REALITY 



Slide 20 
FAULTS 

..DIFFICULT TO WRITE 



Slide 20a 
REALITY 

..SO ARE TESTS AND BUMS 



Slide 21 
FAULTS 

. .Difficult to write 
. .TURN OFF TEACHER 



Slide 21a 
REALITY 

..So are tests and exams 
. .BUT TURN (Xi STUDENTS 
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Slide 22 
FAULTS 

. .Difficult to write 
. .Him off teacher 
. .OFTEN LOW LEVEL 
. COGNITIVE SKILLS 



Slide 22a 
REALm 

..So are tests and exams 
..But turn on students 
. .SO ARE MANY EXAMS - BUT 
OBJECTIVES COVERING 
MORE COMPLEX COGNITIVE 
PROCESSES SUOi AS 
ANALYSIS AND SYNTHESIS 
CAN BE DEVELOPED 



Slide 23 
FAULTS 

. .Difficult to write 

. .Him off teacher 

..Often low level 
cognitive skills 

. .DON'T REALLY IHCLWE 
THE TOTAL MESSAGE 
(CREATIVITY, ADAPT- 
ABILITY, WHOLE CITIZEN, 
RESPONSIBLE, ABILITY 
TO THINK, ETC.) So 
IMPORTANT TO A CCMPLETE 
WELL ROUNDED EDUCATION 



Slide 23a 
REALITY 

..So are tests and exams 
..But turn on students 
..So are many exams - but 
objectives covering 
more complex cognitive 
processes such as 
analysis and synthesis 
can be developed 
. .PERHAPS - Bl/T HOW DO WE 
EVALUATE STUDENT SUC- 
CESS UNLESS HE IS ABLE 
TO DO SOMETHING TO 
DB40NSTRATE THAT HE 
(APPRECIATES, UNDER- 
STANDS, BLOSSOMS OUT) 
AS A RESULT OF THIS 
MAGICAL, MYSTICAL, 
COVERT SOMETHING 



CHAPTER II 



SEQUENCING AND CLUSTERING 
BEHAVIOURAL OBJECTIVES 



L. G. Tippett* 



Recent developments in adult basic education (BTSD) and vocational 
training have placed raany local training institutions such as technical 
institutes and others in the position of having to develop new programs to 
meet the needs of changing pc^lations and training objectives. 

This change in needs in adult basic education has been brought about 

(1) because of the advent of Canada Maiq>ower Training Programs (BTSD) and 

(2) because traditional approaches to deliver adult basic education based 
on the school system model have failed rather miserably. There is need for 
a new content in programs » particularly based on early experiences , a new 

f rocess of learning geared to the particular characteristics and of the 
earner \Axo is peculiar to these programs. 

This change, and the growing demand to deliver adult basic education 
in increasing amounts has caused many adult training personnel in Canada 
to be given the assignment: "Design a new curriculun". 

Initially, this, is received with excitement and ^thusiasm. Almost 
universally however, after the excitement has taipered with reality, the 
question asked is, •'Where do we start?" 

One quickly realizes that the job of designing a curriculum involves 
a process and content * The process being how you design the curriculum and 
the content being \Aiat curriculum and related materials you plan to eventu- 
ally produce. This thai will be a discussion of the process of designing a 
curriculum. This process, as experiaice has shown, can be applied to 
designing a curriculun in any aspect of education and training. An mpOTt- 
ant realization however, is that any model, or process will work if properly 
planned and applied. The Training Research and Development Station and 
others have chos^ the DACUM Model . 



* Head, Career Development Programs 
Trair^-ng Research and Development Station 
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The other aspect, as mentioned, of curriculum design is what to pro- 
duce - leading to a need to define the term "curriculum" itself. Without 
discussing some conventional usages, the Training Research and Development 
Station defines the curriculum as "What the student experiences". This 
definition is chosen to ensure that the curriculum model provides an 
integration of design features vihidx all logically point to the student who 
is taking the courses. This inplies that the instructor is an integral 
part of the design and in fact is the link between the course objectives 
and the student. The instructor designs part of the curriculum - the most 
important part - the learning activities. 

The curriculum design model (process), therefore, must be one which 
provides for a hi^ degree of congruity between what is planned to happen 
and what does happen > 

Several other factors in choosing a model are: 

1. it should be siiH)le. 

2. it should be staged (or phased) so that clearly 
defined jobs can be determined and planned. 

3. it should be easy to communicate to the design 
group and course users outside the design group. 

4. it should be adaptable to the use of behaviorally 
stated objectives and have a built-in sequencing 
and clustering capability. 

5. it should provide easy "fall out" of student pro- 
gress evaluation, course evaluation, in-service 
training of instructors, certification levels and 
other aspects. 

6. it should provide administrators with the capabil- 
ity of planning for the design resources which will 
be required in amount and kind, including personnel. 

7. it must have the capability to produce a curriculum 
\Aich will be an effective and efficient vehicle to 
change the behavior of the target group. 



The DACUM Model 



DACUM sinqply stands for Designing a Curriculum . This model was 
developed in 1969 in a project operated under the auspices of the Social and 
Human Analysis Branch, Department of Regional Econc»nic Expansion and coord- 
inated by Howard L. Clement, a consultant with the branch at the time. 



1 DACUM - Designing a Curriculum, 1969, Department of Regional Economic 
Expansion, Ottawa. 
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The model has been widely used in the development of vocational training 
classes since its publication. Its principles can be applied equally well 
to basic education curriculum. 

2 

The model consists of six phases. A list of these is: 

Phase 1 Defining Terminal Behaviors 

Phase 2 Selecting Evaluation Techniques 

Phase 3 Defining Instructional Unit Objectives 

Phase 4 Designing Instructional Unit Evaluation 

Phase 5 Preparing Learning Activities 

Phase 6 Preparing Individual Student Activities 

A clarification of these six phases will outline the process of 
designing a curriculum based on the model. 



PHASE I: DEFINING TERMINAL BEFIAVIORS 



Generally speaking, the terminal behaviors should present to the 
student a profile of the bdiavior he is expected to displ^ at the end of 
the training period. Iliese are bdiaviorally stated, someidiat "global*' 
objectives. An exanple in mathematics migjit be: 

'The student will be able to read, write, perform the 
four primary operations and solve routine word problems 
using vAiOle numbers." 

Obviously, this is a very general statement of behavior i^ich neither 
defines a scope and sequence of learning objectives, nor identifies the 
condition of knowledge and skill required. 

It would be difficult to justify the design of a training program at 
this point, however, without a broader rationale in which to enfced the 
terminal behavior profile. Therefore, to carry out Phase I conpletely, one 
nust back up to the initial concept of a training program. 

The curriculum designers would be faced with these decisions: 

WHY - a training need has been determined by labour 
market or other demand. 

WHO - is to be trained. Considerable data should be 

collected on the target population so that suitable 
starting points, instructural strategies and other 
parameters can be realistically detemined. 



2 Slightly adapted from the original. 



ERIC 



20 



NH/VT - is to be taught. **What** should be a statement of 
the kind of behavior required at the end of the 
training. It nay be as wide as 'To qualify for a 
level II certificate** or be more narrowly defined 
as **A coimunication, mathematics and science 
program yAiich will provide requisite knowledge 
and skill to take advantage (of a particular 
follow-up program such as trade training)/* 

At this point, a hierarchy of objectives should be knooi such as is 
defined for the Training Research and Development Station LINC Program 
(Learning Individualized for Canadians) 



LINC PROGRAM OBJECTIVES 



LINC OMtlNICATIONS 
COURSE OBJECTIVES 




(TERMINAL ffiHAVIORS) 

INSTRUCTIONAL (ENABLING) 
OBJECTIVES 



LINC MAlieiATICS 
COURSE OBJECTIVES 




(TERMINAL BEHAVIOkS) 

INSTRUCTIONAL (ENABLING) 
OBJECTIVES 



Phase I then involves the decisions related to determining and writing 
the program objectives, course objectives and the terminal bdiaviors. At 
tills point, the OACIM charting technique can be initiated. In this tech- 
nique, the terminal behaviors are arranged in the order (if sequence is 
necessary) ti^t a student would progress through the course. 

The Phase I diart, therefore, would look something like the following, 
each block representing a terminal behavior and in most cases, defining a 
unit of instruction. The actual objective (tenninal behavior) may be 
written in the box or pexliaps sisttply a title of the unit such as •Wiole 
Numbers". 



PiA)lished by Infoimation Canada, Prince Albert, Saskatchewan. 
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DACUM CHART XYZ COURSE 






r 1 




1 1 
1 1 




1 1 

1 1 




• I- 1 
1 1 
1 1 



HOW-TO TRAIN • is a determination of the instructional 
strategy. In many cases, the "how" will 
be in part determined by the external 
constraints of budget, class size, plant 
facilities, etc. 

WHEN-TO TRAIN - is a determination of program scheduling 
based on demands of the labour market 
and needs of the target population. The 
designer should plan for a continuous 
intake of students. 

WHERE-TO TRAIN - is a determination of the geographical 
area best suited to the labour market 
and the target population. 

Phase I, in sunmary, then is a complete planning phase for the new 
training program. Most importantly, a fairly well defined profile of the 
end of training behaviors is determined at this point. Note that the deter- 
mination of program objectives, course objectives and terminal behaviors has 
not been elaborated here. This is extremely difficult in many cases and 
requires considerable organizational effort to organize advisory comnittees, 
design, distribute, collect and analyze the results of questionnaires. Also, 
machinery must be established to determine the behavioral profile desired at 
the end of the course. Without this effort, you may have a good course but 
it may not be a^ all relevant as to what is required. 



PHASE II: SELECTING EVALUATION TECHNIQUES 



Terminal behaviors, being general, and extranely difficult to evaluate 
objectively, do not lend thanselves to the design of actual testing instru- 
ments at this stage. Th? evaluation techniques, however, must be determined 
at this point. 
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Phase II is the point where we determine how to measure achievement of 
the earlier defined behavior profile. In basic education, we usually opt 
for a paper and pencil type evaluation systm. At this stage in the LINC 
Program for exanple, a decision was made to use written tests for each unit 
in two forms, and a diagnostic-placanent test to be prepared for each unit. 
It was further decided to prepare tests in two forms for each sub-unit and 
to set performance criteria in all Form I tests at 1001 accuracy. Also the 
name designation and form numbering system was decided upon for the total 
evaluation system. Student Progress charting and progress accounting 
systems should also be determined at this stage. 



PHASE III: DEFINING INSTRUCTIONAL UNIT OBJECTIVES 



As seen in the hierarchy of objectives outlined in Phase I, these are 
the enabling objectives whjch, if achieved, will produce the termi^nal 
behaviors. These are the .nstructional objectives - the inmediate,, well 
defined, properly sequenced and clustered objectives - on which we base 
learning activities. These objectives should be written, as characteristics 
of good behavioral objectives, in student terms. In the LINC Program, these 
instruction units are called Items and clusters of instructional objectives 
are grouped into these itans. The items are then sequenced to provide en- 
abling experiences which lead to the achievement of the terminal behavior as 
described in Phase I. The sequenced and clustered objectives could be 
charted like this: 



DACUM CHART XYZ COURSE 



[ 



PHASE IV: DESIGNING INSTRUCTIONAL UNIT EVALUATION 



At the end of Phase III all objectives have been clearly defined and 
our evaluation techniques have been determined. Phase IV then involves the 
actual design of measuring instruments. In Phase IV of the development of 
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LINC all Unit Progress Checks, I tan Progress Checks and Placement Inventor- 
ies were prepared. Each objective is checked with one or more test items, 
valid with respect to the objectives and is hopefully, a reliable measure- 
ment of the behavior desired. The use of behavioral objectives, of course, 
improves validity of test items because "what you want to measure" is clearly 
stated. Validity and reliability tests can be done after sufficiently large 
numbers of students have used the test instruments. 

The DACUM chart at the end of Phase IV might look like this. The 
diamonds represent evaluation points of the type: ^ diagnostic and 
placement test for a unit; ^ , an evaluation of mastery of a group of 
instructional objectives (an item) and ^ , a longer term more conprehensive 
test covering several groups of instructional objectives and measuring 
achievement of a terminal behavior. 







DACmi CHART XYZ COITRSE 
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In sumnary, at the end of Phase IV, we know what objectives are to be 
met and we have techniques and instruments to measure achievement of these 
objectives. The "what to teach" decisions are all behind us at this point. 
Our efforts can now be focused on the instructional phases. Instruction 
requires resources. The next phase in the process is preparing these. 



PHASE V: PREPARING LEARNING ACTIVITIES 



One quickly realizes that preparing instructional material, particularly 
in multi-media is extremely difficult, time consuming, and costly. Most 
curriculum designers, therefore, search the commercial market for suitable 
materials, evaluate them and select those which meet the needs of the partic- 
ular course. The instructional objectives from Phase IV provide a good base 
of content on which to base their decisions. Only after one has evaluated 
and selected as much commercial material as possible should the preparation 
of materials begin. At this point, all possible gaps have been filled an^ 
local design of materials can be focused at points where commercial materials 
either do not exist or are not satisfactory. 
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In summary then, this phase involves: 

1. Evaluating existing or available material. 

2. Selecting materials. 

3. Designing and producing materials • (Described in Chapter VI.) 



PHASE VI : PREPARING INDIVIDUAL STUDENT ACTIVITIES 



This is the instructional phase. The instructor matches the student's 
particular needs with the objective and the previously prepared learning 
activities. These learning activities should be based on previous knowledge 
of the student's strengths and weaknesses. Using this model, all circum- 
stances are removed except the learning process itself. The student is able 
through the DACUM chart and the instructional objectives to be fully aware 
of where he is going and the type of performance expected. The instructor 
can then perform as a "manager of learning" in directing the learning 
activities and selecting sound learning experiences with a high degree of 
learner involvement. A prescribed learning experience in this phase would 
involve a statement of: 

1. the objective behavior. 

2. the learning resources. 

3. the learning activity. 

4. an evaluation check on the effect of the activity with respect 
to the objective desired. 

This model has proven to be an effective instrument in curriculum design 
for many. A model, however, can become a master instead of a tool. If it 
becomes a master, one is likely to stray from the main objectives of design- 
ing a learning system which is to achieve some type of behavior change in the 
student. Curriculum designers, therefore, must use such tools pragmatically 
and wisely. This model, or any model, is simply a method of planning and 
controlling a curriculum project. It is not the end product. Effective and 
efficient training of peq)le is. 
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CHAPTER III 



GENERIC TRAINING OBJECTIVES FOR BTSD 



A. D. Smith* 



INTRODUCTION 
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Generic skills are those overt and covert behaviors i^ch are 
ftndamental to the perfbznance of mny tasks and sub tasks carried 
out in a wide range of occupations. Ihey are basic to both special- 
ized applications and job specific skills. 

Generic skills include many of tlie concepts and skills generally 
referred to as academic skills such as mathonatics or comuiications. 
They also include the reasonir^ skills idiich are required to solve 
problems, make decisions, and handle certain areas of discretion. In 
relation to the areas already mentioned, generic skills also involve 
interpersonal skills, lAdch are required lAen working with other 
people, and manipulatiCTi skills which allow one to perfoim on or with 
animate and inanimate objects while using and co-ordinating the hod/ 
senses and muscles. 

Exanples of cleric skills in each of the four areas mentioned 
above could include: 

1* Academic skills - multiplying whole nuiift>ers and reading 
to deteimine job requirements. 

2. Reasoning skills - scheduling woric and diagnosing work 
problens. 

3. Interpersonal skills • giving and receiving rewards and 
discipline. 

4. Mmipulation skills - hand/eye co-ordination and the use 
of proper body posture for lifting and carrying. 

Excluded from generic skills are knowledge and procedural items 

"Manager, Skill Development Studies 
Training Research and Development Station 

An extract from a report Generic Skills pttlished in 1973 by the 
Training Research and Development Station. 
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which are job or ein)loyer specific and which may be attained as part 
of the orientation to a new job. 

The aim of this chapter is to specify the purpose and content 
for generic skills training V^kages for use in Basic Training for 
Skill Development. These will be discussed under the following 
headings: 

I. Purpose of Training Packages . 

II. Content and Organization of Training Packages . 

The following exponents are to be prepared as a result of the 
development work outlined in this chapter: 

1. A discrete curriculum package for each unit. These packages 
*are to include: 

a. the instructional objectives 

b. diagnostic and placement tests 

c. references by objectives to appropriate learning 
materials vdiich have been selected or developed, 
and 

d. achievement tests. 

2. An instructor 's manual which includes information on the 
background, objectives, con5X)nents, individualized process 
and procedures needed to iii5)lement and operate the program. 

3. An answer book for all diagnostic, placement and achievement 
tests. 

4. Individualized learning packages for those items viiere suit- 
able or y/Aiere sufficient learning materials are not available 
in published materials. 



I. PURPOSE OF TRAINING PACKAGES 



The training packages are to be designed to facilitate achievement 
of the following objectives: 

1. To assist students to achieve the specific generic data 
skills which are, required for chosen occupations. 

2. To allow students to gain recognition for skills attained 
outside the educational system. 

3. To provide remedial academic upgrading for those \Axo have 
the official prerequisites but have forgotten the actual 
skills or knowledge. 



ERLC 



27 



4. To enable continuous intake and exit of students. 

5. To provide a process by which sidults may identify and 
acquire additional generic data skills required to ad- 
vance their careers. 

It is foreseen that the training packages will be inplemented 
by occupational training institutions in a variety of situations ^ such 
as: 

1. Using the program to prepare students for occi?>ational 
training courses or for job entry. 

2. Using the content of the program as a vocational guide to 
help students establish appropriate occupational goals. 

3. Integrating the program with occupatio]\al training courses 
by allowing training time daily for the development of 
the generic data skills. 

4. Unifying this program with occupational training courses 
by treating the data concepts and the occupational skill 
requirements as a unit. 

It is also foreseen that the training packages will be used by 
enployers: 

1. In conjunction with OfTP Training on the Job (TOJ) programs. 

2. In conjunction with cOTipany on job training (OJT) programs. 

3. To facilitate career programs by en;>loyers. 

4. To permit recognition of data skills learned by experience 
on the job. 



II. cmmr of training packages 



The content to be covered by the training packages is shown in 
the following: 

1. The scope and content are shown in the skill content 
chai't attached as Table 1. 

2. A sample fom of an occupational profile is included 
as Table 2. 
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TABLE 1: COiTEOT CHART 



UNIT 




ITEM A 




ITEM B 




%ITEM C 




ITEM D 




ITEM E 




ITEM F j 


1 1 . WHOLE 
1 NIMBERS 


1 


Read, Write 
^ Count 




Add and 
Subtract 




Multiply 




Divide 




Word 
Problcas 




Round Off 


1 2. FRACTIONS 


1 


Read $, Write 




Add and 
Subtract 




Mjltiply 




Divide 




Word 
Problcas 






1 3. KCINALS 


1 


Read t, Write 




Add and 
Subtract 




Miltiply 
^ Divide 




Word 
ProbleiFs 




Round Off 




4. PERCENT 




Read ^ Write 




Calculate 




Ratio 




Proportion 




Word 
Problems 




1 5. NIXED 

1 OPERATIONS 




Equivalents 




Order of 
Operaticms 




Ocaq»ation 
Word Problems 




Quick 

Calculations 






1 6. ESTIMATE 
1 MEASURES 


1 


Ti«e 


Weis^t 




Distance 




Area 




Liquid 
Voluw 




SkMtial 
Voluae 


1 7. MEASURE 


1 


Scales 




Time 




Weight 




Distance 




Quantity 




Verniers 


1 8. METRIC 
1 MEASURE 




Weight 




Distance 




Quantity 




Weight 
Conversion 




Distance 
Conversion 




Quantity 

Conversion 


9. DIMENSIONS 




FbTMS & 

Figures 




Draw/Sketch 




Read Scale 
Drawings 




Measure 
Scale Drawings 




Drw to 
Scale 






10. DIMENSIONS | 


Ptoriiaeter 




Area 




Voliae 




Read 
Graohs 




Construct 
Graphs 




11. ALGEBRA 




One 

Variable 




Variables 




Quadi.tics 




Variations 






12. TRIG 




Logs 




Slide Rule 




Calculation 






13. LISTEN 




Literal 
Ccnprehension 




Interpretive 
Comprehension 




Evaluative 
Conprehension 


14. TALK 




Directions/ 
Instructions 




Infomation 




Questions 




FonMl Talks 




15. READ 




Literal 
CGMprehenslon 




Interpretive 
^ Evaluative 




Conparative 
Evaluation »1 




Books 




GoB^arative 
Evaluation #2 




Comparative 
Evaluation #3 


16. BUSINESS 
READ 




Notes, Letters 
( MeMos 




Forms 


1 

1 


Charts 




Manuals 




17. MRITE 1 




Phrases 




Sentences 




Paragraphs 




Poiws 




Mans 




Letters 


1 IB. HRITE 1 
1 REPORTS 1 




Infoimtion 
Reports 




Recomendati^n 
Reports 




Sort 




Classify 




'Mmical 
Reports 




If. EXCHANGE 
IDEAS 




Debate 




Peer 
Debate 




Supervisor 
Debate 




Confermces 






20. DIAGNOSE 




List Causes 




Look Up 
Causes 




Set 

Priorities 




Look Up 
Methods 




Probing 
Questions 




Use Senses 


21. UTITUDE 




Sii^>le 

Instructions 




Con^lex 
Instructions 




Instructions 




Set 

Priorities 





34 



29 



ocojpaticn: paikter 



TABLE 2: S;WPLE (XTUPATIONAL PROFILE 

CCDO NO. 



878S-110 



codes : %/ requi red 
o 0 optional 

ERIC 



UNIT 




ITEM A 




ITEM B 




ITEM C 




ITEM D 




ITEM E 




ITEM F 


1 1 . WHOLE 
1 IVlfVtKa 








Vkl /d 
St^ract 












Nbr^ 
l*^lcns 




Ro^^ m 


1 2. FRACTIONS 
1 




KcatklAritc 
If 




Sduract 




^hAl/pJy 

V 




QiJde 
V 




Wordf 
l^i^lcns 






3. DECIHALS 




HcHllAriti' 
V 




Ul /.d 




^1^1 lie 




Wonj/ 
IhfWicws* 




Rq^Off 




4. PERCENT 




i^c.id /Write 
V 




i.iii-jn.ito' 
V 




V 




Ihrqn/tion 
V 




PV*>c«s 




5. MIXED 

OPERATIONS 




Ix^^Vlllcnts 




Ordci^f 
< "pc^vi ions 




(VcupaMon 
noruu^roD 1 CHS 




giiick / 
(Uilcl^tions 




1 6. ESTIMATE 1 
1 MEASURES 1 


V 




hci/»t 
V 




If 








Liq/d 




Spatial 
Volunc 


1 7. MEASURE 




V 




V 




I^it 








Qi^ity 




Verniers 


1 8.. METRIC 
1 MEASURE 




IVcijiht 




Dist.incc 








hVight 1 
Conwrsion J 


Distvice 
Conversion 




Quantity 
Conversion 


9. DIMENSIONS 


1 


I lyiros 




Hr.i^f^ctc'h 




Kc:ki/:;iU> 
MrJ/nK^ 




^lL*-asiire/ 
Scalc^p^inRS 




nr» to 
Scale 




ID. DIMENSIONS 
















Kr.td 
Graphs 




ConstrtJct 
(iraphs 




11. ALGEBRA 




Van .ihU* 




luo 

\.iri.ibles 




(Xia'idriit lis 




\annt tons 






12. TRIG 








^1 nil' Klill 




( •iUiil.it loii 






13. LISTEN 




( onif)wiLNis ion 




IntcrpiVt ivt* 
tOfnpi^y^*ns ion 




1 valiLit^ c 
romp i|y|k>ns ion 




14. TALK 




IhrcctyBns/ 
lnst»t iot)s 




Infonwt ion 

\r 




'jm^^^fis 




l-omil Talks 






15. READ 




Literal/ 
('ofltprUlofi^ i on 




Intcrpr^ ivc 
Ivu^itivi' 




iimpanij^w 
Ivalixynoft "1 








(>)aipar^ive 
l*valuxion '2 




C'x)«iparaMve 
livali^non *y 


16. BUSINESS 
READ 




Notes, uftters 




sr 




\r 




^lani^s 








1 17. WRITE 












raruc^nh^ 














1 18. URITE 
1 REPORTS 




Inforn^ ion 




Ucconmcf^tion 




Sor/ 

✓ 




Classy 




Technical 
Reports 






19. EXCHANGE 
IDEAS 








Vlct/ 




Siipcrwsor 




(bnfci^ccs 






2D. DIAGNOSE 












Set / 
Pri softies 




Ixwk lA 




I'rohWifi 
Qu||fions 




Use ^J^ses 


21. LATITUDE 




Simple/ 
Instinct ions 




Inst fusions 




Instni^ions 




Set / 
IVic^^ics 
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3. A matrix of occupations and skills is attached as 
Annex A. 

4. The instructional objectives for the items included 
are at Annex B. 

It is is^rtant to realize that the content has not been arranged 
in a learning hierarchy. In essence, the preceding work has been an 
identification of data skills idiich are coomon to many occupations 
and a further level of identifying their actual use by the ocaq>ations 
which were analyzed. Ihe evidence contained in this report must ue 
carefully studied before making any assunqptions about sequenced skill 
development. Many of the items will have to be designed as discrete 
skills to ensure that students do not have to attain extraneous skills 
based on a traditional sequence of learning. 

Ihe instructional packages are to be developed for a target popu- 
lation ^0 are at least fiinctionally literate. Adults vdio can achieve 
a measured Grade IV in the reading con^xment of standard tests such 
as the Canadian Test of Basic Skills should be able to achieve the re* 
quired skills included in the instructional packages. 

The objectives at Annex B to this report do not necessarily in- 
clude the enabling skills required by adults ^o are unable to achieve 
these behaviors. It will be recalled that judgement was 
exercised in the design of the data collection instninent, in the 
belief that a number of enabling or subordinate skills could be iden- 
tified on the basis of the functional skills carried out by the workers. 
Data was not collected on skills such as spelling requir^nents, diction, 
use of prefixes and suffixes - nor was data collected on knowledge items 
such as mathematics nomenclatures, (numerator, denominator, subtralumd, 
etc.), purpose of nouns. Verbs, adverbs and prepositions. The develope^^s 
will therefore necessarily have to identify and build into the curri- 
culum two types of enabling objectives: 

1. Subordinate skills which can reasonably be considered 

to be subsumed by the objectives at Annex B. For example, 
the skill of writing letters necessarily iiq)lies some 
ability to spell words. 

2. Prerequisite skills ^ich students necessarily have to 
attain before they would be able to learn the instructional 
objectives at Annex B. For example, students ^o are learn- 
ing elementary arithmetic skills will probably have to learn 
the meanings of certain symbols and words. 

Caution must be exercised in the selection and in the use of these 
additional curriculum objectives. Because these are to be inserted on 
the basis of judgment rather than on empirical evidence, the develop- 
ment staff should restrict their use to those which are an obvious 
requirement and be extremely careful when using them for diagnor^tic or 
achievement tests. Ve have, for example, evidence that most of the 
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occupations surveyed have to be able to multiply and divide fractions 
but we have no evidence to suggest that they have to be able to cor- 
rectly identify the words numerator and denominator » even though the 
use of these skills may help develop the skills of handling fractions. 

From the preceding paragraphs it will be seen that the develop- 
ment staff will necessarily have to prepare or develop the following: 

1. Enabling objectives designed to help those who cannot 
perform the instructional objectives. 

2. Subordinate objectives %Aich can be subsumed from the 
instructional objectives. 

3. Test items to be used for diagnostic ^ prescription and 
achievement purposes. 

4. A master sequential learning model to ensure that students » 
who need to achieve any or all of the required skills ^ 
learn them in an appropriate order or sequence. 

To fulfill the specified objectives ^ the developed program should 
enbrace the following types of individualization: 

1. Individualized Content . The skills to be attained by 
each student to be based on his occtqiational goal, with 
institution and/or student options, as suggested in the 
appropriate occupational profiles. 

2. Individualized Process . Continuous intake and exit of 
students to be easily administered. Students to be given 
necessary diagnostic and prescription tests to determine 
skill development needs and be allowed ^en reasonable 
and possible, a wide selection of learning materials in 
various media. 

3. Individualized Pacing . Students t6 be allowed to develop 
skills at a pace consistent with their learning abilities. 
Achievement tests to be capable of individual or small 
groap administration based on the appropriate time to 
meet student rather than administrative needs. 

Many of the exercises and tests should involve operant as well as 
respondent behavior. For exanple, it is more iiqportant that a worker 
be able to state and siqpport a conclusion reached from oral or written 
material than be able to identify a correct conclusion. Certainly, 
factually correct responses are required at the literal coii;>rehension 
level but operant behavior becomes important at the interpretive and 
evaluative levels of conprehension. 

As previously mentioned, the content has not been ordeted or 
sequenced - nor have the mathematics and comrnmications units been 
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separately classified by these titles. This will allow the developers 
to integrate the learning experiences so the students can learn and 
apply mathematics and conimmications concepts rather than having them 
separated according to sid)jects. 

The processes used in LINC (Learning Individualized for Canadians) » 
will be used for the development of these training packages. 
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INSTRUCTIONAL OBJECTIVES 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 
IfHT 1 - WHOLE NUMBERS 



ITEM A - WIT 1 

READ. WRTIE. AND COM QWDLE NOMBERS) 
OBJECTIVES 

1. count a set of tp to 100 objects and state or write the number of 
objects counted. 

2. Identify place value of digits in a given lAole nunfcer with up to 
7 digits. 

3. Read a givoi whole nunber less than 10 million and write its name 
in words. 

4. Write a idiole nunber less than 10 million given the word name of 
the nunber. 

5. Write nunbers to one million after counting or reading the word 
name of the nunber. 

6. Arrange a set of idiole nuii>ers in ascending or descending order. 
ITCM B - ItHT 1 

AED AND SUBTRACT (WDIE NIMBERS) 
(fflJECTIVES 

1. Perfonn additions of whole nunbers given tvistructions such as: 

Miat is the total of 8 and 5? 
Find the sum of 8 and 5. 
Add 8 and 5. 
8 plus 5 is 
8 ♦ 5 - . 
8 

♦5 

2. Find the sun of a set of ^le numbers of 1-5 digits including 
cases lAere carrying is required. 
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ANMHX B 
INSTRUCTION/M OBJECTIVES 



3. Perform subtraction of \ihole numbers given instructions such as: 

8 minus 5 is . 
Subtract 5 frciTB'. 
Find the difference of 8 and 5. 

If you had 8 and gave away 5, you would have . 

8 - 5 » . 

8 
^5 

4. Find the difference of two 1-5 digit nuni)ers including cases i^re 
borrowing is required. 

5. Solve a routine word problem using subtraction or addition of lAole 
nuinbers. 



ITEM C - UNIT 1 
MULTIPLY (WHOLE NUMBERS) 



OBJECTIVES 



1. Perform multiplication of \A)ole nuinbers givai instructions such as: 

6 times 2 is . 

Three two^s are . 

The product of 6 anH' 2 is 

6 X 2 « . 

6 
x2 

2. Find the product of a given pair of ^ole nunbers of lAich one number 
is up to 5 digits and the other number \xp to three digits. 

3. Solve a routine word problem using multiplication or division of 
^ole numbers. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 



ITEM D - WIT 1 
DIVIEE (WHOLE NIMBERS) 



OBJECTIVES 



1. Perform division of two simple vdiole numbers (no remainder) given 
instructions such as: 

2 divided by 6 is 

2 into 6 goes times. 

Hie quotient of 6 divided by 2 is . 

6 t 2 - 
2)6 • 

6 - "7" 

7 — 

2. Find quotients of 1-5 digit dividends and 1-3 divisors including 
cases lAere there are remainders. 

3. Solve routine word problems using division of i^ole numbers. 

ITEM E - UNIT 1 

WORD PROBLEMS (WK)LE NU^ERS) 



OBJECTIVE 



1. Use addition, subtraction, multiplication or division, as appropriate, 
in the solution of routine occupational word problems with t^ole 
numbers. 



ITEM F - UNIT 1 

ROWD OFF (WHOLE NUMBERS) 



OBJECTIVE 



1. Round off a nunber to nearest ten, hundred, thousand, ten thousand, 
or hundred thousand. 



48 



43 



ANNEX B 

INSTRUCTIONAL OBJECTIVES 

UNIT 2 - FRACTIONS 

NB: Fractions used for test items are to be those 
''hich could realistically be expected to be 
usc^ in occupational work (e.g., one to two 
digit denaninators OR even hundreds, thousands). 
Fractions used in occupational operations generally 
have denominators such as 2,3,4,5,10,25,100,1000, 
2000 and not 7,9,37,1005, etc. Comparison or 
reduction of fractions usually involves no more 
than three fractions and the dencaninators usually 
have a measurement relationship such as x/4, x/8, 
x/16, or x/9 and x/27. 

ITEM A - UNIT 2 

READ AND WRITE (FRACTIONS) 

OBJECTIVES 



1. Identify a common £7iction as being a number of parts of a total 
number of equal parts. 

2. Reduce a fraction to its simplest form (lowest terms). 

3. Change improper fractions to mixed numbers and mixed numbers 
to improper fractions. 

4. Virite equivalent fractions for a given fraction given a new 
numerator and denominator. 

5. Arrange a set of fractions and/or mixed numbers in ascending 
or descending order. 

6. Write vAxat fractional part one number is of another in simplest 
form. 



ITEM B - UNIT 2 

ADD AND SUBTRACT (FRACTIONS) 

OBJECTIVES 



1, Calculate the stm in siirplest form of two or three common fractions 
with like denominators. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 



2. Calculate the sum in sin?)lest form of two or three common 
fractions with unlike denominators. 

3. Calculate the sum in sinplest form of two or three mixed 
numbers with like and unlike denominators. 

4. Calculate the sum in simplest form of two or three numbers 
which include at least one proper fraction and one mixed 
nunber. 

5. Calculate the difference in sinplest form of two common 
fractions with like denominators. 

6. Calculate the difference in simplest form of two common 
fractions with unlike denominators. 

7. -alculate the difference in sinplest form of two mixed nunbers 
which have like denominators including cases where borrowing 
is required. 

8. Calculate the difference in siiiplest form of two mixed nunbers 
which have unlike denominators including cases vAere borrowing 
is required. 

9. Solve a routine word problem using addition and/or subtraction 
of fractional nunfcers. 



ITEM C > UNIT 2 
MULTIPLY (FRACTIONS) 



OBJECTIVES 



1. Calculate the product in simplest form of: 

a. a conmon fraction and a >^ole number 

b. two coinnon fractions. 

2. Calculate the prodiict.in siJH)lest form: 

a. a mixed number and a whole number 

b. two mixed numbers. 

3. Calculate the product in simplest form of two or three numbers 
which include at least one cornnon fraction and one mixed number. 
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ANNEX B 
INSTRUCriQN/VL OBJECTIVES 



4. Solve a routine word problem using multiplication of fractional 
nmhers. 



ITEM D - UNIT 2 
DIVIDE (FRACTIONS) 

OBJECTIVES 



1. Calculate the quotient in sin5)lest form of: 

a. a common fraction divided by a vrtiole nunfcer 

b. two coninon fractions 

c. a vtfiole number divided by a common fraction. 

2. Calculate the quotient in sin5)lest form of: 
a. a mixed nionber and a \ihole nuaber 

h. a mixed nund)er and a comnon number 
c. two mixed nurrbers. 

3. Solve a routine word problem using division of fractional numbers. 



ITEM E " UNIT 2 

WORD PROBLEMS (FRACTIONS) 

OBJECTIVES 



1. Solve a routine word problem using addition, subtraction, multi- 
plication and/or division of fractions. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 
UNIT 3 - DECIMALS 

NB: Decimals used for test items are to be those vAich 
could realistically be expected to be used in occu- 
pational work (e.g., no more than three zeros re- 
quired to position the decimal (.0001) or no more than 
three digits after the decimal is rounded off (1.782). 

ITEM A - UNIT 3 

READ AND WRITE (DECIMALS) 

OBJECTIVES 

1. Read a given 1-4 place decimal fraction and write its name in words. 

2. Write a 1-4 place decimal fraction given the word name of the number. 

3. Arrange a set of decimal fractions in ascending or descending order. 

4. Write: 

a. the total value of a given combination of dollars, quarters, 
dimes, nickels, and cents. 

b. the combination of dollars and coins equivalent to a given 
money value. 



ITEM B - UNIT 3 

ADD AND SUBTRACT (DECIMALS) 

OBJECTIVES 



1. Calculate the sum of a set of decimal fractions. 

2. Calculate the difference of two decimal fractions. 

3. Solve a routine word problain using addition and/or subtraction of 
decimal fractions. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 



ITEM C ■ UNIT 3 

MULTIPLY AND DIVIDE (DECIMALS) 



OBJECTIVES 



!• Calculate the product of two decimal fractions when rounding off 
is (a) not required (b) required. 

2. Calculate to a given number of decimal places, the quotient of a 
vrfxole nimnber divided into: 

a. a whole number larger than the divisor. 

b. a whole number smaller than the divisor. 

c. a decimal fraction. 

3. Calculate to a given number of decimal places, the quotient of a 
decimal fraction divided into: 

Fi ia decimal fraction . 
b. a whole number. 

4. Solve a routine word problem using multiplication and/or division 
of decimals. 



ITEM D - UNIT 3 

WORD PROBLEMS (DECIMALS) 



OBJECTIVE 



1. Solve a routine word problem using addition, subtraction, multipli- 
cation, and/or division of decimal fractions and conmon fractions 
or mixed numbers. 



ITEM E - UNIT 3 
ROUND OFF (DECIMALS) 



OBJECTIVES 



1. Round off a decimal fraction to the nearest tenth, hundredth, 
thousandth, or ten- thousandth. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 
UNIT 4 - PERCENT 

ITEM A - UNIT 4 

READ AND VVRITE (PERCENT) >, 

OBJECTIVES 

1. Identify a percent as equivalent to a fraction with 100 as the 
denominator . 

2. Identify the I symbol by writing in words a given percentage. 

ITEM B - UNIT 4 
CALCULATE (PERCENT) 

OBJECTIVES 

1. Calculate a percent of a number. (Type 1: a% of b = .) 

2. Solve a routine word problem involving calculation of a 
pfc/cent of a number. 

3. Calculate the percent one rmber is of another. (IVpe 2: 
a is % of b) . 

4. Solve a routine word problem involving calculation of the 
percent-^one rajntf)er is of another. 

5. Calculate a number yAien a percent of it is known. (Type 3: 
a% of » b). 

6. Solve a routine word problem involving calculation of a 
number \Aien a percent of it is known. 

7 Solve percentage word problems involving money, people, objects, 
time, and other data. 
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ANNEX B 
INSTKilCTIONAL OBJECTIVES 

ITEM C - UNIT 4 
RATIO CPERCENT) 

OBJECTIVES 

1. Identify comparisons of two nunfcers in the forms a:b, a, is to b", 

F 

or "a parts to b parts", etc., as ratios. 

2. State or write in simplest form (lowest terms) a ratio of two nuirbers 
in the forr' 

a. a:b 

b. a 

3. Write a ratio given a description of the two nuiribers one of which must 
be determined by adding or subtracting the nuirbers in a given ratio. 

ITEM D " UNIT 4 
PROPORTION (PERCENT) 

OBJECTIVES 

1. Identify an equality of two nunfcers in the form: a:b - c:d, § = j> 

"a is to b as c is to d", or "a parts to b parts is the same as 
c parts to d parts", etc., as a proportion. 

2. Calculate the unknown number given 3 numbers in a proportion. 

3. Calculate a number when a fraction of it is known, using the propor- 
tion method. 

4. Solve a routine word problem which involves either a ratio or a 
rate, using the proportion method. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 

ITEM E - UNIT 4 

WORD PROBLEMS (PERCEOT) 

OBJECTIVE 

1. Solve a routine word problem which involves the use of percentage 
calctdations, ratio or proportion. 

UNIT 5 - MIXED OPERATIONS 

ITEM A - UNIT 5 

EQUIVALENTS (MIXED OPERATIONS) 

OBJECTI'.^S 

1. Convert: 

a. a common fraction or mixed nuni)er to a deciiiial fraction. 

b. a decimal fraction to a common fraction or mixed nmrber. 

2. Convert: 

a. a percent to a fraction. 

b. a fraction to a percent. 

3. Convert: 

a. a percent to a decimal. 

b. a decimal to a percent. 

4. Use standard tables to convert: 

a. a fraction to a percent. 

b. a percent to a fraction. 

ITEM B - UNIT 5 

ORDER OF OPERATIONS (MIXED OPERATIONS) 



OBJECTIVES 



1. Simplify an expression involving addition and subtraction of 
v^ole numbers or fractions. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 



2. Simplify an expression involving addition and subtraction of \A\ole 
numbers or fractioizs. 

3. Sinplify an expression involving addition and/or subtraction and 
multiplication and/or division of ^ole numbers or fractions. 

4. Sin5)lify . expression ^ich requires the removal of parentheses. 

5. Sin5)lify an expression in which the numerator and/or denominator 
of a fraction is a sum, difference, product, or quotient of two 
whole nimfcers. 

6. State the order in vtfiich order of operav^jns rules apply. 

7. Write a mathematical expression relating the facts in a given 
problem and then siinplify that expression. 



ITEM C - UNIT 5 

OCCUPATION WORD PROBLEMS (MIXED OPERATION S) 



OBJECTIVE 



1. Given word problems appropriate to a selected occiq^tion, be 
able to: select arithmetic functions to be performed; the 
oslci' in \Aich they are to be done; and obtain the required 
solution. 



ITEM D - UNIT 5 

QUICK CALCULATIONS (MIXED OPERATIONS) 



OBJECTIVE 



1. Given appropriate occupational arithmetic calculation require- 
ments the student will be able to perform the operations using 
common short cut methods. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 
UNIT 6 - ESTIMATE MEASURES 



NB: The ability to estimate measurements is rarely included 

in academic programs. A high proportion of the occupations 
surveyed indicated that this is a requirement. It is also 
to be noted that many yAxo require this skill do not also 
have to have the skill of actually measuring. The following 
criteria are appropriate to this item: 

(1) Excessively small or large measurement estimates are not 
required. Extremes such as seconds - months, less than an 
ounce - over 100 lbs., less than a quarter inch - over 50 
ft., less than one fluid ounce - over 10 gallons, should 
be avoided. 

(2) Objects used should have a regular configuration of a cir- 
cular, square or rectangular type. 

(3) Students should be able to use various sensory organs, feel, 
see, handle, etc. It would not be realistic, for example, 
to ask a student to judge the weight of a sealed container 
without being able to lift it. Cognitive knowledge of the 
coDHposition of various materials is not a requirment, e.g., 
hardwood is nonnally heavier than softwood. 

(4) Few workers require all of the itans in this unit and each 
itm should therefore be treated independently of the other 
items. 



ITEM A - UNIT 6 

TIME (ESTimTF. MEASURES) 



OBJECTIVES 

1. Identify seconds, minutes, hours, days, weeks, months and years 
as units of time. 

2. Convert units of time to other units (e.g., 90 minutes is 1 1/2 
hours) . 

3. Be able to estimate the duration of time actually required to 
perform routine functions, within 50% of the actual time. 

NB: TTie routine functions selected should not be for 
excessively short or long durations. 
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ANNEX B 
INSTOIICTIONAL OBJECTIVES 



ITEM B - UNIT 6 

WEIOfT (ESTIMATE MEASURES) 



OBJECTIVES 



1. Identify ounces, pounds and tons as units of weight measure. 

2. Convert units of weight to other units (e.g., 32 ounces is 2 
pounds). 

3. Be able to estimate the weights of coinnon objects within 20% 
of the actual weight. 

NB: The common objects should be items v4iich the student 
could be expected to apply judgment by handling and 
not be excessively light or heavy. 



ITEM C - UNIT 6 

DISTANCE (ESTimTE MEASURES) 



OBJECTIVES 



1. Identify inches, feet, yards and miles as units of distance. 

2. Convert units of distance to other units (e.g., 18 inches = 
1 1/2 feet). 

3. Be able to estimate the length, width or height of conmion 
objects or the distance between objects, within 20% of the 

:tual distance. 

NB: The objects or distance between objects should not 
be excessively short or lengthy distances but should 
be distances over which the student could be expected 
to apply judgment by visual inspection. 
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AfJNEX B 
INSTRUCTIONAL OBJECTIVES 



ITEM D - UNIT 6 

AREA (ESTimTE MEASlHESj 



OBJECTIVES 



1. Identify square inches, square feet, square yards as the measure 
of the surface of a figure. 

2. Convert units of area to other units (e.g., 144 square inches 
equals one square foot). 

3. Be able to estimate the area of conmon objects within 50% plus 
or minus of the actual area. 

NB: The coirnion objects should be rectangular or square and 
not be excessively small or large. 



ITEM E - UNIT 6 

LIQUID VOLUME [ESTIMATE MEASURES) 



OBJECTIVES 



1. Identify fluid ounces, pints, quarts and gallons as units of 
liquid capacity. 

2. Convert units of liquid measure to other units (e.g., 1 1/2 
gallons equals 6 quarts) . 

3. Be able to estimate the liquid contents or the capcity of 
conmon containers, within 50% plus or minus of the actual 
content or capcity. 

NB: The coirmon containers to be regular shaped bottles, 
tins, etc., of a circular, square or rectangular 
configuration without irregular constrictions except 
for the neck and not be excessively large or small. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 

ITEM F - UNIT 6 

SPATIAL VOLUME (ESTIM/VTE MEASURES) 
OBJECTIVES 

1. Identify cubic inches, cubic feet, and cubic yards as units 
of space measurement. 

2. Convert units of space measuranent to other units (e.g., 27 
cubic feet equals one cubic yard) . 

3. Be able to estimate the cubic measurement of a COTmon object 
or space, within 50% plus or minus of the actual size. 

NB; The common object or space to be a reasonable size (small - 
as a book to large as a classroom) and to be square 
or rectangular in configuration. 

UNIT 7 - MEASURE 

ITEM A - UNIT 7 
SCALES (MEASURE) 

OBJECTIVE 

1. Visually read coiranon scales, such as thermaneters, weigh scales, 
clocks, water meters, gas meters, tape measures, pressure scales 
and odometers. 

ITEM B - UNIT 7 
TIME (MEASURE) 

OBJECTIVES 

1. Add and subtract 12-hour clock times including cases in subtractiwi 
where borrowing is required. 
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annd: b 

INSTRUCTIONAL OBJEaiVES 

2. Be able to measure the actual duration of time expended to 
perform routine functions, correct within 5% of actual 
duration. 

ITEM C - UNIT 7 
IVEIGHT (MEASURE) 

OBJECTIVES 

1. Add and subtract weight measures including cases where borrowing 
is required. 

2. Be able to weigh common objects on balance or spring loaded 
scales, correct witiiin one unit of the scale readings. 

ITEM D - UNIT 7 
DISTANCE (MEASURE) 

OBJECTIVES 

1. Add and subtract units of distance including cases where borrowing 
is required. 

2. Measure a given distance or one dimension of an object with a 
ruler or tape measure to the nearest 1/8 inch. 

ITEM E " UNIT 7 
QUA^f^ITY (MEASURE) 

OBJECTIVES 

1. Add and subtract units of liquid measure including cases vdiere 
re-grouping is requiied. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 

2. Be able to measure liquid content or capacity of a container 
to within 1 unit of the scale on the measuring instrument. 

ITEM F - UNIT 7 
VERNIERS (MEASURE) 

1. Usually read, from pictorial representations of common verniers, 
the indicated measurement to the nearest unit. 

UNIT 8 - METRIC MEASUREMENT 

ITEM A - UNIT 8 

WEIOTT (METRIC MEASUREMENT) 



OBJECTIVES 



1. Identify milligrams, centigrams, decigrams, grains, decagrams, 
hectograms, kilograms and metric tons as units of metric weight 
measure. 

2. Convert units of weigjit from one metric unit to another (e.g., 
100 grams equals one hectogram) . 

3. Be able to estimate the weight of common objects within 20% plus 
or minus of the actual metric weight. 

4. Be able to weigh coiranon objects on balance or spring loaded 
scales, correct within one unit of the metric scale readings. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 

ITEM B - UNIT 8 

DISTANCE (METRIC MEASUREMENT) 

OBJECTIVES 

1. Identify millimeters, centimeters, decimeters, meters, 
decameter, hectometers and kilometers as units of metric 
distance measure. 

2. Convert units of distance from one metric unit to another (e.g., 
one hectometer equals 100 meters) . 

3. Be able to estimate the length, width or height of comnon 
objects or the distance between objects, within 20% plus 
or minus of the actual distance. 

4. Be able to measure a given distance or one dimension object 
with a ruler or tape measure to the nearest centimeter, 

ITEM C - UNIT 8 

QU/WTITY (METRIC MEASUREME^f^) 



OBJECTIVES 



1. Identify milliliter, centiliter, deciliter, liter, decaliter 
and hectoliter as units of metric capacity. 

2. Convert units of capacity frcmi one metric unit to another (e.g., 
10 liters equals one decaliter) . 

3. Be able to estinate in metric units the contents or the capacity 
of common containers, within 20% plus or minus of the actual 
content or capacity. 

4. Be able to measure content or capacity of a container to within 
1 unit of the scale of the measuring instrument. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 

ITEM D - UNIT 8 

WEIGHT CONVERSION (METRIC MEASUREMEm') 
OBJECTIVES 

1. Given tables, be able to convert metric weight measurements 
to English measurements. 

2. Given tables, be able to convert English weight lAeasurements 
to metric measuronents . 

ITEM E - UNIT 8 

DISTANCE CONVERSION (METRIC MEASUREMENTS) 
OBJECTIVES 

1. Given tables, be able to convert metric distance measurements 
to English measurements. 

2. Given tables, be able to convert English distance measurements 
to metric measurements. 

ITEM F - UNIT 8 

QUAOTITY CONVERSION (METRIC MEASUREMENT) 
OBJECTIVES 

1. Given tables, be able to convert metric measures of capacity 
to English measuronents . 

2. Given tables, be able to ccmrert English quantities to metric 
measurements. 
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ANNEX B 
INSTRUCTIONAL C3JECTIVES 
UNIT 9 - DlfE^SIONS 



ITEM A ' UNIT 9 

FIGURES AND FOl^ (DIMENSIONS) 



OBJECTIVES 



1. Identify rectangles, squares, triangles, circles, rectangular 
solids, cubes, pyramids, cones, cylinders and spheres from a 
set of objects, pictures of objects, or names of objects. 

2. Identify: 

a. rectangles, squares, triangles, and circles as plane 
figures (t'A'c-dimensional). 

b. rectangular solids, cubes, pyramids, cones, cylinders, 
and spheres as solid figures (tliree-dimensional) . 

3. Identify rectangles, squares, triangles, circles, rectangular 
solids, cubes, and cylinders given their definitions or 
characteristics. 



ITEM B - UNIT 9 
L.WSKETCH (DIMENSIONS) 



OBJECTIVES 



1. Draw or sketch (not to scale) (student choice) squares, rec- 
tangles, triangles, circles, cubes, rectangular solids, 
cylinders, cones and pyramids. 

2. Represent, by drawing or sketching (student choice), coninon 
objects using oblique projections. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 

ITEM C - UNIT 9 

READ SCALE DRAWINGS (DIMENSIONS) 
OBJECTIVES 

1. Identify 1:4, 6:1, 1" to the foot, 1" » 5 mi., 1:50,000, etc., 
as examples of scales used to represent reductions or expansions 
of the actual size of an object. 

2. Convert a scale measurement to true size given the scale. 

3. Identify a given drawing as a map, blue print, idiotograph, 
schematic diagram or assembly (exploded view) drawing. 

4. Trace a particular subsystem on a single schematic drawing. 

5. Identify the order in \^ich parts are put together in an 
assembly drawing. 

ITEM D - UNIT 9 

MEASURE SCALE DRAWINGS (MEASUREMENTS) 
OBJECTIVE 

1. Calculate actual dimensions of an object given a ruler and a 
simple scale drawing of the object. 

ITEM E - UNIT 9 

DRAW TO SCALE (MEASUREMENTS) 

OBJECTIVE 

1. Given a compass, right angle triangle and a ruler be able 
to draw to scale front, top and side representations of 
comnon objects ^ich are rectangular solids, cylinders, 
cones or pyramids. 
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ANNEX B 

Instructional objectives 



NB: For items B to E, enphasis should not be placed on drafting 
skills (line weight, comer construction, etc.) or on artis- 
tic skills (shading, balance, etc.) but on accuracy of 
reading, measuronents to and from, etc. In other words, 
can the student properly read the drawings and construct 
sketches or drawings vJiich can be accurately read? 

UNIT 10 ' DIMENSIONS 

ITEM A. " WIT 10 
PERI^CTER (DIMENSIONS) 

OBJECTIVES 

1. Identify perimeter as the length of the outer boundary of a 
plane figure and identify the units used as inches, feet, yards, 
etc. 

2. Calculate the perimeter of a given rectangle or square. 

3. Measure the perimeter of a given figure by successive ruler 
measurements. Identify as a constant equal to c/d for all 
circles and identify its approximate value as 3 1/7 or 3.14. 

4. Calculate the circumference of a circle given: 
a» its radius 

b. its diameter 

5. Solve a routine word problem involving perimeter. 

ITEM B ' UNIT 10 
AREA (DIMENSIONS) 

OBJECTIVES 

1. Identify area as the measurement of the surface of a figure 
and state units used. (sq. in., sq. ft., sq. yd., sq. mi.) 
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ANNEX B 
INSTRUCTICm OBJECTIVES 

2. Calculate the area of a given square. 

3. Calculate the area of a given rectangle • 

4. Calculate the area of a given triangle. 

5. Calculate the area of a given circle. 

6. Solve practical problems involving area calculations of squares , 
rectangles 9 triangles, and circles. 

ITEM C - UNIT 10 
VOLUME (DIMENSIONS) 

OBJECTIVES 

1. Calculate the volume of a given rectangular solid (including 
cubes) . 

2. Calculate the volume of a given cylinder. 

3. Solve practical problems involving volume calculations of 
rectangular solids and cylinders. 

ITEM D - UNIT 10 

READ GRAPHS (DIMENSIONS) 

OBJECTIVE 

1. Read specified data from sinple line, bar and circle graphs. 

ITEM F - UNIT 10 

(CONSTRUCT (aRAPHt: (DIMENSIONS) 

OBJECTIVE 

1. Giver* siitple data construct line, and bar graphs to display 
a visual representation of the data. 

er|c 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 
UNIT 11 - ALGEBRA 

ITEM A - UNIT 11 

ONE VARIABLE (ALGEBRA) 

OBJECTIVES 

1. Write single variable open sentences given verbal statemaits 
of relations. 

2. Solve single variable open sentences. 

3. Solve routine word problans vdiich require eqijations in one unknown. 

ITEM B - UNIT 11 

TWO VARIA BLES (ALGEBRA) 

OBJECTIVES 

1. Write two variable open sentences given verbal statements of 
relations . 

2. Solve two variable open sentences. 

3. Solve routine word problems which require equations in two unknowns. 

ITEM C - UNIT 11 
QUADRATICS (ALGEBRA) 

OBJECTIVES 

Not establirhfed. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 

ITEM D - UNIT 11 
VARIATIONS (ALGEBRA) 

OBJECTIVES 

Not established. 

UNIT 12 - TRIGONOMETRY 

ITEM A - UNIT 12 
LOGARI'mMS (TRIGONOMETRY) 

OBJECTIVES 

1. Solve multiplication problems by using logarithm tables. 

2. Solve division problems by using logarithm tables. 

ITEM B - UNIT 12 

SLIDE RULE (TRIGONONCTRY) 

OBJECTIVES 

1. Solve multiplication problems by using a slide rule. 

2. Solve division problans hy using a slide rule. 

3. Solve exponent and roo;t problems by using a slide rule. 

ITEM C - UNIT 12 
CALCULATION (TRIGONOMETRY) 

OBJECTIVES 



Not established. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 
UNIT 13 - LISTEN 



NB: The audio tapes should contain a variety of simulated 
occupational listening conditions. 



ITEM A ■ UNIT 13 

LITERAL COMPREHENSION (LISTEN} 



OBJECTIVES 



1. Listen to an audio tape and be able to answer **vdiat**, *\Ao", 
'•where**, and '•vdien" questions. 

2. Listen to an audio tape and be able to identify the order or 
sequence of required activities. 

3. Listen to an audio tape and be able to identify the latitude 
allowed by the speaker. 

4. Listen to an audio tape and be able to perform a series of 
activities that requires recall of instructions or information. 

NB: The students to be encouraged to take notes during 

the playback and to be aware of the objectives (but not 
the actual test itans) . This item is intended to develop 
the skills of effectively listening for the factual aspects 
of oral conrunication. 



ITEM B - UNIT 13 

irffERPRETIVE OOMPRHiENSION (LISTEN) 



OBJECTIVES 



1. Listen to an audio tape and be able to identify the main purpose 
of the talk. 

2. Listen to an audio tape and be able to answer 'Tiow" and 'Wiy" 
questions . 

3. Listen to an audio tape and be able to answer questions i^ich 
require a generalized interpretation of the talk. 
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ANNEX B 
INSTRUCTION/VL OBJECTIVES 

ITEM C ■ UNIT 15 

EVALUATIVE COMPREHENSION (LISTEN) 
OBJECTIVES 

1. Listen to an audio tape and be able to identify speakers^ 
statements as being facts or opinions. 

2. Listen to an audio tape and be able to perform a series 
of activities that requires recall of instructions or 
information AND interpretive and evaluative comprehension. 

NB: I tans B and C: operant as well as respondent behavior 
are iin)ortant in these two items. It should not be 
expected that students achieve conformity of conclusions, 
rather it should be expected that they be able to defend 
or support their interpretive and evaluative conpre- 
hension. 

UNIT 14 - TALK 

ITEM A - UNIT 14 

DIRECTIONS/ INSTRUCTIONS (TALK) 

OBJECTIVES 

1. Organize and express directions in a logical sequence and with 
time limitations, given direction content. 

2. Answer questions orally \Aich require interpretation of an 
experience . 

ITEM B - UNIT 14 
INPORM/VTION (TALK) 

OBJECTIVES 

1. Repeat information which has been told. 

73 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 



2. State the correct answer to questions on a subject about vdiich 
information has been given. 

3. Describe orally, in a few short sentences, a given person, place, 
thing, or job performed. 

4. Answer questions orally about material read (such as a note, 
mono or letter). 

5. Describe an experience or same event that was witnessed (or 
read about). 

6. Explain to scaneone else how to perform a fairly co!!?)l ex task, 
given experience at performing the task, or having learned 
frcMU another person or a manual how to perform the task. 



ITEM C - UNIT 14 
QUESTIONS (TALK) 



OBJECTIVE 



1. Given a short statement of a typical and realistic occupational 
problem and a coach or peer who has a list of facts pertinent 
to the problem each student will be able to obtain at least 51% 
of the facts by asking information seeking questions. 

NB: ItCTis A and B are intended to develop skills of inparting 
infomation and instructions. The exercises and tests 
should be carefully designed to ensure that memory or 
understanding is not being tested. In item C the ability 
to obtain facts or opinions by questioning, rather than 
the ability to frame "proper** questions, is the essential 
skill. 



ITEM D > UNIT 14 
FORMAL TALKS (TALK) 



OBJECTIVE 



Not established. 
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ANNEX B 
INSTRUCTICNAL OBJECTIVES 
UNIT 15 - READ 



NB: Memory is not involved in these object ives. The 
written mateiial (up to Grade level 8.5 Fry Read 
ability Scale) supplied to the student should be 
available to the student during performance. 



ITEM A ' UNIT 15 

LITERAL COMPREHENSION (READ) 



OBJECTIVES 



1. Identify statements as being true or false after reading a 
given selection. 

2. Answer '\Aat", "vAo", "irfiere", and '"idien" questions using 
factual information from a given selection. 

3. Given a set of instructions involving activities to be 
performed by a number of people, and the particular role, 
prepare a list and sequence of activities to be performed. 



ITEM B - UNIT 15 

INTERPRETIVE AND EVALUATIVE COMPREHENSION (READ) 



OBJECTIVES 



1. Answer 'Vhy" and "how" questions using factual information 
from a given selection. 

2. Answer questions yhidi require a generalized interpretation 
^ich can be inferred from facts in a given selection. 

3. Identify causes and effects from a given selection. 

4. Identify and support a conclusion that can be inferred 
from a set of fact*; or from a given selection. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 

5. Identify and support a prediction based on information 
given in a selection. 

6. Separate facts and opinions in a given selection. 

7. G^ven a set of instructions involving literal, inter- 
pretative and evaluative conq)rehension, and a statement 
of the current situation, prepare a list and sequence 
of activities to be performed. 

ITEM C - UNIT 15 

COMPARATIVE EVALUATION #1 (READ) 
OBJECTIVE 

1. Conpare two or more written directions and identify discrepancies. 

ITEM D ' UNIT 15 
BOOKS (READ) 

OBJECTIVES 

1. Be able to locate a telephone nunfcer, address or business 
firm from a telephone directory. 

2. Be able to locate an item and its cost and catalogue nunfcer 
from a catalogue. 

3. Be able to obtain the correct spelling for a word from a 
dictionary. 

4. Be able to locate a particular reference in a reference 
book or company manual. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 

ITEM E - UNIT 15 

a}MPARATIVE EVALUATION »2 (READ) 
OBJECTIVE 

1, Be able to locate similar infomation from two or more 
catalogues, books or manuals and determine idiich one is 
most suita^^le based on given criteria such as price, delivery 
date, colo. , size, etc. 

ITEM F - UNIT 15 
COMPARATIVE EVALUATION »3 

OBJECTIVE 

1. Be able to locate similar information from two or more books, 
catalogues or manuals, set criteria and determine ^ich one 
is the most suitable on the criteria established. 

UNIT 16 - BUSINESS READ 

ITEM A - UNIT 16 

NOTES, LETTERS OR MEMOS (BUSINESS READ) 
OBJECTIVE 

1. Given notes, letters or memos, be able to identify: 

a. the author 

b. the recipient 

c. the activities required of the recipient 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 



ITEM B - UNIT 16 
FORMS (BUSINESS READ) 



OBJECTIVE 



1. Given a variety of comnon business forms such as vrork orders, 
job orders, issue and receipt vouchers, claims and purchase 
orders, be able to answer factual questions based on the data 
on the forms. 



ITEM C - UNIT 16 
(HARTS (BUSINESS READ) 



OBJECTIVE 



1. Given comnon reference tables and charts such as organization 
charts, parts charts, reference tables, including charts and 
tables designed in matrix form (rows and columns), be able 
to answer factual questions based on the data supplied. 



ITEM D - UNIT 16 
MWUALS (BUSINESS READ) 



OBJECTIVE 



1. Given a typical company manual or union agreement containing 
administrative, policy, personnel and financial rules and 
regulations, be able to answer factual questions based on 
the manual for items such as hours of work, rates of pay, 
vacation or other leave, grievances, pranotion and discipline. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 
UNIT 17 - WRITE 



ITEM A - UNIT 17 
PHRASES (WRITE) 



OBJECTIVE 



1. Given single data in the form of words and figures the 

student will be able to write phrases yAiich are legible and 
vdiich convey the desired information. 



Um B - UNIT 17 
SENTENCES (WRITE) 



OBJECTIVE 



1. Given single data in the form of words and figures the 
student will be able to write ccanplete sentences. 

NB: The data obtained indicates that sane occupations 
only write by filling in phrases and sentences on 
forms. Therefore, the only criteria for items A 
and B is viiether or not the phrase or sentence would 
be ui^derstood. Correct use of capital letters, punc- 
tuation, spelling, tense, etc., although desirable, 
is not critical at this level of performance. 



ITEM C - UNH 17 
PARAGRAPHS (WRITE) 



OBJEaiVES 



1. Given simple data in the form of words and figures the student 
will be able to write sentences using capital letters, punctua- 
tion, modifiers and spelling correctly. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 



2. Write a short article or report of four or five paragraphs 
using proper sentence structure, word usage, punctuation, 
and paragraph form, on a topic of interest. 



ITEM D - UNIT 17 
FORMS (WRITE) 



OBJECTIVES 



1. Given typical employment application forms the student will 
be able to complete them correctly, neatly and legibly. 

2. Given typical business forms such as work orders, job orders, 
issue £u»i receipt vouchers , claims , and purchase orders and 
the necessary data in words and figures, the student will be 
able to correctly coi^jlete or fill in the appropriate portions 
of the forms. 



ITEM E - UNIT 17 
MEMOS (WRITE) 



OBJECTIVES 



1. The student will be able to set up the parts of a business 
mesio in proper form, sender's name and/or appointment, data, 
addressee's name and/or appointment, subject heading, body 
and signature. 

2. Compose business memos in proper form using good sentence 
and paragraph structure and proper punctuation, from infor- 
mation supplied. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 



ITEM F - UNIT 17 
LETTERS (WRITE) 

OBJECTIVIS 

1. The student will be able to set up the parts of a business 
letter in proper form, (sender's address, date, addressee's 
address, salutation, body, conplementary close and signature). 

2. Given the information for a letter the student will be able to 
organize the body of the letter into a logical sequence, such 
as the following: 

a. An introduction \A\ich describes the purpose or aim of 
the letter (why it is being written) . 

b. A background or history so the reader can understand 
the discussion which follows. 

c. The discussion necessary for the reader to know the 
current facts on the arguments presented. 

d. A closure vrfiich includes the recommendation, decision, 
action requested or stminary. 

3. Gonqwse business letters in proper form, using good sentence 
and paragraph structure, logical sequences, proper punctua- 
tion and spelling, from information supplied. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 
UNIT 18 - WRITE REPORTS 



ITEM A - UNn 18 



INFORmTION REPORTS (WRITE REPORTS) 
OBJECTIVE 

1. Given real or similated occupational conditions in vihich the 
student can use a variety of his senses the student will be 
able to prepare a written factual report vdiich is correct, 
concise and reasonably ccmplete. 

ITEM B - UNIT 18 

RBCC> »<ENDATION REPORTS (WRITE REPORTS) 
OBJECTIVES 



1. Given a typical and simple type of occupational problem the 
student will be able to select alternative co^irses of action 
to alleviate or solve the problem. 

2. Given a typical and sinplt type of occupational problem and 
a list of alternative Co^'ues of action to alleviate or 
solve the problem the student will be able to list advantages 
and disadvantages for each alternative. 

3. Given real or s inula ted ocaq;>ational problems in vhich the 
student can use a variety of his senses , the student will 

be able to prepare & recomniendation report idiich includes al- 
ternatives, courses of action to alleviate or solve the 
problem and the advantages and disadvantages of each. 

NB: The condition for items A and B could be an actual 
activity or a film. The students should be aware of 
asty facts \i\ich would already be known to the recipient 
of the report so that he would only have to report v*iat 
his siqpervisor needs to know. The problems selected 
must be of a nature where the student could be expected 
to apply judgment on the problan and the alternatives. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 



ITEM C - UJIT 18 
SORT (WRITC REPORTS) 



OBJECTIVii 



1. Given data or objects and a classification criteria the 

student will be able to sort the data/objects by the various 
classifications. 

NB: The data/objects and the classification criteria 

should require a minimum of judgment. The ability to 
sort rather than the ability to apply judgment is 
the essential criteria for this objective. The 
exercises and rests for this objection and for the 
following items should gp beyond the idea of sorting 
for filing, such as sorting for warehousing, sorting 
tools and materials, sorting figures and vouchers. 



ITEM D - UNIT 18 
CLASSIFY (WRITE REPORTS) 



OBJECTIVE 



1. Given data or objects such as receipts and expenditures 
or pictures of a variety of objects i^ich can be readily 
classifi*^ by different lists, the student will be able 
to set up a classification system and sort the data/objects 
by the various classifications. 

NB: The data/objects should be presented in a way that 
there is an apparent need to sort them for various 
reasons such as filing, warehousing, providing in- 
formation. The student should be aware of ••why" he 
has to classify them and the desired skill is "how to 
class if/'. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 

ITEM E - UNIT 18 

TECHNICAL REPORTS (WRITE REPORTS) 
OBJECTIVES 

Not established. 

UNIT 19 - EXCHANGE IDEAS 

ITEM A - UNIT 19 
DEBATE (EXCHANGE IDEAS) 

OBJECTIVES 

1. TTie student will be able to withhold judgment \4iile listening 
or reading. 

2. The student will be able to clarify point*, not 'jp-iefftoo^ as 
a re~ul* of listening or reading. 

3. The student will be able to apply judgjnent to material heard 
or read after he has understood the other's intent, and then 
state his reaction in oral or written form. 

4. The student will be able to state his point of view in oral 
and written form and defend the point in subsequent conmuni- 
cation. 

NB 1: Although these objectives suggest a high degree of 
skill the tests and learning processes shotJ-d be 
at an unsophisticr ced level. Preferably tlie same 
material (oral and written) should be used for each 
objective so the student will learn to systematically 
follow these steps. 

NB 2: Interpersonal behaviors necessavv to supplement these 
skills will be identified as pare of the "people" 
analysis to be carried out at a later date. At that 
time, the hi^ier level skills of persuading and nego- 
tiating will be examined. 

NB 3: The above objectives are only intended for the one to 
one level of comnunication. 
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ANNEX B 
INSTRUCriON/VL OBJECTIVES 



ITBt B - UNIT 19 

PEER DEBATE (EXCHANGE IDEAS) 



ORTECTIVES 



1. The student will be able to identify the fact that, in oral 
or vnritten conversation with customers, workers' camnents, 
points of view and ideas are accepted as ccnspaxty policy and 
that the company for which he works will probably morally 
and legally have to COTiply with any comnitments which the 
employee makes. 

2. Given case studies of conversations between customers and 
employees the student will be able to decide: 

a. Whether to give the customer a final decision and ' 
what the decision should be, or; 

b. Ask the supervisoi to make the decision. 

3. Given case studies of discussions between fellow workers the 
student will be able to decide: 

a. Whether to compromise, accept the viewpoints of the 
other, or retain his own viewpoint and then state his 
decision in oral or written form, or; 

b. Ask the supervisor to make the decision. 



ITBi C - UNIT 19 

SUPERVISOR DEBATE (EXCHANGE IDEAS) 



OBJECTIVES 



1. The student will be able to identify the fact that, in a 
debate between worker and sup*»rvisor, the worker is normally 
expected to seek clarification si5)ervisor's instructions 
and to make the final decision to compromise, accept or reject 
the enployee's point o^ view. 

2. The student will be able to perform objectives 1 to 4 of 
Item A, in oral form, with a person v*io is, or appears to 
be, a person of authority. 
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ANNEX B 



INSTTUJCTIQNAL OBJECTIVES 



ITEM D - UNIT 19 
CONFERENCES (EXCHANGE IDEAS) 



OBJECTIVES 



1. In a conference situation with peers and a person of 
authority the student will be able to: 

a. State his point of view or supply factual information 
\^en asked. 

b. Contribute to the discussion by: 

(1) building on previous discussion; and 

(2) introducing single concepts at a time. 

c. Take notes for any future activities irfiich are his 
responsibility. 

NB: Uhits A, B and C are intended for the one to one hivel 
of communication. In unit D the skills are those of 
a participant in a conference and not those of the 
chairman or supervisor. Business conferences rarely 
take the foxm of a "meeting" of the type that involves 
makiMk motions and other parliamentary procedures. 
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ANNEX B 



INSTRUCTIONAL OBJECTIVES 



UNIT 20 - DIAOWSE 



ITEM A - UNIT 20 
LIST CAUSES (DIAGNOSE) 



(BJECTIVES 



1. Given a series of facts pertinent to a single problem situation 
composed of statements i^ich are either symptoms (effects of) 
or problems (causes of) the student will be able to correctly 
id^tify the statements as syn;)toms or problems. 

2. Given a series of facts pertinent to a simple problm composed 
of statements vMch are either synf>toms or problems, and 
statements vrfiich are neither, the student will be able to 
correctly identify the synptom and pioblem statements. 

3. Given a series of symptom facts pertinent to a simple problem 
the student will be ablo to prepare a list of the possible 
problems . 



ITEM B - UNIT 20 




\ 1. Given a series of syrapton facts pertinent to a simple 

problem and ^ chart or written material which dekcribes / 

1 conmon cause - effect relationships, the student ;will be 
able to prepare a list of the possible problems. \ 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 



ITEM C - UNIT 20 

SET PRIORITIES (DIAGNOSE) 



OBJECTIVES 



1. Given a series of symptam and problem facts pertinent to 
a simple problem situation vrfiich has a cause - effect 
relationship between two or niore symptoms, the student 
will be able to trace the symptom - problem relationship. 

2. Given a list of possible syii5)toms and problems for a 
simple problem and their relationship the student will be 
able to establish a priority list for problem identification. 

3. Given a series of synptom facts pertinent to a simple 
problem situation the student will be able to identify 
possible problems and establish a priority list for 
problem identification. 



ITEM D - UNIT 20 

LOOK UP METHODS (DIAGNOSE) 



OBJECTIVE 



1. Given a single problem situation, with: the possible synpti^ 
and problems identified; the possible symptom/problem 
relationship established; a list of priorities for problem 
identification; and an instructional manual which describes 
how to check for faults; the student will be able to identify 
the appropriate portions of the manual. 
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ANNEX 3 
INSTRUCTIONAL OBJECTIVES 



ITEM E ■ UNIT 20 

PROBING QUESTIONS (DIAGNOSE) 



OBJECTIVES 



1. Given a simple problem situation in which possible cause- 
effect relationships can be identified by questioning and 
a list of possible problems with their priorities, the 
student will be able to eliminate in order possible problems 
by asking probing questions until the problem h?^ ' en iden- 
tified. 

2. Given a simple problem situation in which possible cause- 
effect relationships can be identified by questioning, the 
student will be able to prepare a list of the possible 
probl ems , establish a prioiity nst and eliminate in order 
possible problems by asking probing questions until the 
problem has been identif ied . 



ITEM F - UNIT 20 
USE SENSES (DIAGNOSE) 

OBJECTIVES \ 1 * ^ 

~ I \ - - 

1. Given a sinqjle probl^^n situation in v^ich possible cause- 
effect relationships ^an be identified by sensory organs 
(toiK:h, see, smell, pi taste) and a list of possible prob- 
lons.witK their priormes, the student will be able to 
elim5.nate in ordy t ft^^ possible problems by the use of his 
sens(^s until tne problem has been identified. 

2. Given a simple proi>le*m situation in vrfiich possible cause- 
effect: relationships can be identified b> sensory organs, 
the student will be able to prepare a list of possible 
problems, establish a priority list and eliminate in order 
possible problems by the use of his senses until the problem 
has been identified. 
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INSTRUCTIONAL OBJECTIVES 
UNIT 21 - LATITUDE 



ITEM A - UNIT 21 

SIMPLE INSTRUCTIONS (LATIUJIE) 



OBJECTIVE 



1. Giveii suiple instructions of the nature of, "If this 

happens - do that", in oral or written form, the student 
will be able to recall and comply with the instructions, 
when appropriate. 



ITEM B - UNIT ?i 

COMPLEX INSmiCriONS (LATITUDE) 



OBJECTIVE 



1. Given instructions of the nature of, "If A and B happen, 
do X, but if A and C happen, do Y", in oral or written 
form the s'.udent will be abl0 to recall and exactly with 
the instructions, yhea appropriate^ 



ITEM C - WIT 2 1 

OOMPOWD INSTRUCTIONS (LATITUDE) 



OBJECTIVE 

I 

1. Given a wo.vk situation and instruction of the ^nature of, 
*'If A and B happen use ycur judgment \^ether to do X, Y, 
or Z", the student will be able to recall the instruction, 
decide on and take appropriate action v^en appropriate. 
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ANNEX B 
INSTRUCTIONAL OBJECTIVES 



I TEM D - IJKIT 21 

SET PRIORITIES (LATITUDE) 



OBJECTIVE 



1. Given a hypothetical working situation , a list of both routine 
and unique tasks to be performed , deadlines vdiich have been 
established for COT?)letion certain tasks » and estimates of 
tine to perform each task, tiie student will be able to establish 
a work priority list. 
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CHAPTER rV 



THE INDIVIDUALIZED LEARNING PROCESS 



V. W. Mullen* 



A. INTRODUCTION 



ERIC 



The Training Research and Developnent Station has developed a program 
in basic education to prepare undereducated adults as quickly as possible 
to enter vocational training and to handle the requiranents of reading, 
i-nriting and arithmetic in everyday life. The program had to cover an 
academic range from about Grade 5 to Grade 10, the level specified by most 
provinces as the necessary prerequisite to training in vocational skills. 
The program, known as LINC: Learning Individualized for Canadians, is now 
being extended to cover the fuller range of skills. 

The program is individualized adult basic education, thus making poss- 
ible a system of continuous intake and exit. Such a system allows adult 
students to enter a program vtfienever they are rea^, and leave without 
waiting for a fixed graduation day. other students can be enrolled to 
fill a training place as soon as it becomes vacant . In that way, there 
can be more effective use of instructors and ciassro(liBs , tcpsts per training 
day can be lowered, and students waste less time in irkirling.'* | 

The LINC program satisfies these needs at Wo iJvels of individualiz- 
ation; it is arranged so that provinces car ada^t it eas.Uy to m^t their 
varying reapirements of standards and content, and its s)|steins ^roach 
provides a^jrocess of complete individualization for continL^us intake and 
exit . 

Basically, its content satisfies the core requirements cf all or most 
provinces, but its process allows content units to be add<»d, subtracted, or 
changed to fit a range of provincial curricula. Its system of prescription 
allows each teacher to include his own favourite textbooks and methods. 
Its system of -evaluation allows criteria at check points to be adjusted to 
fit local standards. Courses other than the mathanatics and canmunications 
of tlie LINC program can be modelled on its process. LINC thus serves as a 
framework on which to build the variations required by the provinces and 
the institutions which provide adult basic education in each province. 

* Chief, Adult Development Division 
Training Research and Development Station 
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B. THE LINC PROGRAM 



The LINC program package may be considered from three points of view: 
the components , the content, and the process . 



Conponents 



The ccMnponents are the three courses: Mathematics, Developmental 
Reading y and Functional Reading. Developmental Reading is concerned with 
teaching the skills of receiving oral and written information and conpre- 
bending 9 interpreting and evaluating it. Functional Reading, on the other 
hand, deals more with transmitting information, mostly by writing, and with 
the usages which make correct writing possible. 



Content 



Behavioural objectives define the content of the progi in a range of 
difficulty from about Grade 5 to Grade 10. To make the individualized pro- 
cess work, objectives are grouped into hcmogeneous concept items , and itans 
in turn are arraiiged in units . Mathematics, for example, has ten units sub- 
divided into a total of 45 itans lAich in turn include 266 learning object- 
ives. (See Table 1: The Organization of a LINC Unit.) 

The objectives make the heart of the LINC program. They keep the units 
internally consistent, and they control the stages of difficulty by their 
grading. All the evaluation instruments - the placanent inventory, the 
short term item checks, and thp longer range unit chedc^ - are based on the- 
itan objectives. Evalualiion and even student process is pressed visually 
ifAen a student crones edch objectiire off his chart afcer he has shown i 
mastery of it at bbth.the item and unit level; the chart of objectives shoi|rs 
both student and instructor wh^t he has done and what remains to be done 
for him to complete the program. Thus progress is rated not in teims of a 
series of percentage nkrks but of objectives, for idiich the criterion has 
been met. Students arfl^ exjjbcted to master every objective in the program; 
some do it quickly aad tothers more slowly. Time, then, and not irfiat fraction 
of a program a student has passed^ an examination on, becomes the individual 
variable, ^ote that content will also be an indivi^al variable in the 
revision now being prepared for data skills.) 

The content of LINC was limited to Mathematics and various aspects of 
Conmunications as core necessities in providing students with at least the 
minimum essentials to qualify to enter vocational training , to get an aca* 
danic certificate that would open doors to jobs which require that level of 
education, and to cope with the practical needs of daily living. Now, for 
the Generic Skills project, LINC is being extended to include the inathema- 
tics and coranunications skills required as part of vocational training. 
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Process 



The process is the system of diagnosis, placement, prescription and 
evaluation which makes possible the individualization of the program, both 
for the student in the cxassroom and for the institution that wishes to 
adapt the program to suit its own requirements. (See Table 2: Individualized 
Process.) 

The process is designed for individualized instruction of students in 
three well-defined steps: 

1. Diagnosis and Placement 

Every unit in the program begins with a placement inventory in 
which there is at least one question on every instructional objective 
in the unit. Students are placed according to the objectives they are 
unable to meet. When a course begins, everybody may take the placement 
inventory for Unit 1 of a course at the same time, but the students 
soon spread out. Some students make mistakes immediately and they must 
be assigned work on the unit. Other students may breeze through several 
placement inventories before they have to stop. Within a short time, 
all the students are working at different levels. As students conplete 
the program or drop out to talce jobs or for other reasons, other students 
take their empty seats in a continous intake-exit situation. These new 
entrants start with the first placement inventory themselves. Thus 
after the first day there is no set time for placement testing; it 
must go on continually. 

Placement is important for adult students who are returning to 
school after a nunfcer of years away. They want to find out what they 
know and what they don't know, what they have completed aiid what they 
still have to finish to reach a certain goal. Placement related to 
objectives forces a student to face the truJi of his functioning level; 
he cannot hide behind a. vague grade level. Placement helps to make him 
responsible for his own* learning, and that is an inportant aspect of an 
individualized program for adults. Furthermore, as he shows mastery of 
certain objectives, either on placement inventories or further evaluation 
checks, he feels a sense of satisfaction and acconplishment. 

At the same time, by diagnosing a student's weaknesses and strengths 
and by placing him correctly, an instructor begins to understand him. 
Placement is the first step in a fuller diagnosis of the student's cap- 
abilities, so that during the whole training program the instructor can 
prescribe the b ist materials and use the best r..ethods to fit the 
student's level and to match his particular learning abilities. 



2. Prescribed Learning Activities 

Before an instructor can prescribe work successfully for an adult 
student, he has to do more than analyze his mistakes on paper. At the 
prescription stage the instructor should sit down with each student, 
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TABLE 2 

Individualized Process 




INSTRUCTIONAL 
RESOURCES 



1. 


Saskitchewan 




NewSUrt Inc. 




prepared and 




reconnended 




material . 


2. 


Other 




comnercial 




textbooks . 




prog, books, 




tapes, films. 




fllmstrl ps. 




labs. kits. 


3. 


Locally 




prepared 




worksheets. 




prog, books, 




tapes, films. 




fi Imstrlps, 




video tapes. 




kits. 


4. 


Other: 




discussions. 




peer teaching. 




lectures . 



rvES 



j±0 



Stb ''t Completed Course 



PI - Placement Inveniory 
IPC - Item Progress Check 
UPC - Unit Progress Check 
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examine the mistakes he has made and relate them to the objectives 
which state what must be learned. The student must learn to face his 
mistakes rnd realize what has to be done to overcome them. 

In this personal contact, the instructor can begin to develop the 
one-to-one relationship which is so important in adult basic education. 
Perhaps personal and social difficulties may be explored at the same 
time as the academic. The instructor may not be able to perform fully 
as a counsellor or to settle the student's life problems at this point, 
but he can at least lend a syii5)athetic ear. The ability to establish 
a good relationship with a student, to show empathy and understanding, 
may be more inportant attributes in an instructor than pedagogical 
skills. 

The instructor's big task in prescribing is matching materials 
and jnethods with an individual's learning needs. Because ir most 
training programs there are adults with different backgrounds of 
knowledge as well as differing interests and capabilities, no single 
textbook or set of materials can satisfy all the individual require- 
ments. Some people learn best in short steps, others in longer ones; 
some grasp new principles quickly, others need more explanations or 
more examples oi* more reinforcement. A set of books aimed at the 
average adult stadent (or, too often in adult programs, the average 
child) can satisfy neither the slow learner at one end of the scale 
nor the fast learner at the other. Some students prefer programmed 
materials, with a chance to check results after every frame; others 
prefer workbooks with lots of drill on every new concept; still 
others feel cheated if they are not given an old- fashion textbook 
such as they may have used in school years before. 

The same observations have been made about methods. One student 
learns best by reading a textual explanation, another by looking at 
a visual presentation or illustration, a third by hearing an explana- 
tion. A student may freeze during a teacher's explanation and not 
remember a word, but he may absorb fully the explanation of a peer 
for vAiom he has no fear, especially if that person has just faced 
the same problan and has oveiCOTie it. Some students still want to 
learn in traditional class groups, with the instructor giving a group 
lesson with assignments based on it. 

Thus the instructor must be prepared to draw on a wide reper- 
toire of materials p'^A approaches in order to match them in his pres- 
criptions for the particular needs of a particular student. This 
means that an instructor must be well acquainted with a good stock 
of resources. He must know their strengths and weaknesses and their 
suitability for different students. He has to be able to orchestrate 
them progressively in his prescriptions, bringing them in at the 
ri^t moment, particularly in reading programs. Finally, if commercial 
materials and the usual methods fail, a good instructor must be capable 
of creating his own materials and methods to zero in precisely on the 
student's problem. 
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The LINC program recannends only a basic set of comnercial 
books for classrocxn use. Each classroom needs only a few copies 
of each since students are working at different levels, perhaps 
on different courses, at one time. Any institution iin)l«nent?Jig 
the LINC prograir may use their own stqjply of materials instead 
of those reconmended or they can add them to the basic set to 
make it more COT5>rehensive. 



3, Evaluation 

The LINC program has three levels of evaluation. At first 
the placement inventory reveals the item and objoctive each 
student must start work on. When a student completes work on all 
the objectives of an item, and both the instructor and the student 
are satisfied that he is ready for the test^ he takes the itan 
progress check immediately. If he riches the criterion score, 
he continues work on further items until he coii5)letes the unit. 
He then takes the unit progress check, tests the objectives 

of all the items in an integrated way. A final evaluation idien 
the st'jdent has coinpleted the course checks the student's reten- 
tion of all the objectives in the course. (See Tables 1 and 2). 

All the item and unit progress checks have two forms. The 
rec^nnended criterion on all Form 1 checks is conplete mastery. 
A mistake indicates the need for a further prescription, prefer- 
ably with different materials or a new method, to give the student 
a fresh opportunity to overcOTie his difficulty. After remedial 
work he tries Fbrm 2 of the check. 

Criterion scores reconmended for Form 2 checks are about 901 
and there is a 'Teacher's Judgment" safety valve to let students 
proceed or do further remedial work in borderline cases . 

Instructors must stress to students that evaluation is for 
diagnostic and remedial purposes and not for the usual percentage 
mark to place them on a scale with other students to see if they 
pass or fail. Gradually both the instructor and the student should 
come to treat evaluation as a source of learning more about the 
student himself, his strengths and weaknesses, and the methods and 
materials best suited for his learning needs. 

The cycle of testing, individual prescription, individual work 
with varying materials and methods, and individual evaluation 
continues throughout the program. 



C. THE INSTRUCTOR IN TOE CLASSROOM 



Training instructors for the LINC progr im includes introducing them 
to a new role in the classioom. This role differs greatly from that of 
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the teacher in the traditional classroan. The LINC process itself, as 
a vehicle for new approaches and a new role for instructors, does, if 
they follow it conscientiously, force them to change their ways. However, 
they can change irwre effectively when they are made conscious of the 
contrast between the role of instructor as a planner of lessons, class 
lecturer, and the role of diagnostician, prescription specialist, personal 
instructor and counsellor, and the classroom manager of systems and materials. 

The LINC program releases an instructor from preparing daily lessons 
to spend more time with students who need his help. He can call on students 
to do peer teaching when he cannot keep up with all the personal demands 
for assistance. There is no better way for a student to reinforce or 
consolidate what he has just ^earned than to teach it to another. There 
can be more than the recommended, 15 adults in a LINC classroom if the 
instructor can organize efficient peer teaching. 

An instructor has to learn to promote a changed physical and social 
environment in the classroOTi. There must be classroom rules, but the 
students themselves ought to participate in making them and in disci- 
plining tJiemselves, They may want to be free to move around the classroom, 
to change the furniture, to sit and work in small groups or individually, 
to break up the straight rows, to help others and get help as they require, 
to smoke or to drink coffee vdiile working. One of the most difficult 
social changes to make in the classroom is to drag the teacher out from 
behind his desk, to stop him from using it as a protective shield between 
himself and his students. Hie successful teacher of adults has to work 
from within the group and not from outside it. 

An anecdote about a new LINC classroom in one of the eastern provinces 
describes an importtuit visitor. He could not distinguish the instructor 
from the students, he was puzzled to find a student working permanently 
at the teacher's desk over against the wall, and he was somev*iat chagrined 
that everybody kept on working and paid little attention to him. His 
conment was that he had never visited a classroom like that before. 

Adults often enter upgrading or retraining programs with built-in 
insecurity and fears: fear of failing again, fear of being laUghed at 
of made to feel foolish, fear that all the other students will do well 
but that they won't measure up, fear of an imposing building and of an 
educational system not really meant for them, and fear of a well-educated 
instructor who is far above thrm. The fears may be irrational, but they 
are nevertheless pre:,ent, and they must be understood. 

Then there are physiological lacks, especially among the economically 
disadvantaged. Students may have failing eyesight or hearing and not be 
conscious of it; they may find it difficult to sit still in a confined 
space for several hours or even to hold a pencil. They may suffer from 
poor health or malnutrition because of unwise eating habits, or from in- 
sufficient sleep because of an irregular life style. Excessive use of 
alcohol or drugs may have impaired their intellectual acuity. 
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Instructors must learn to consider the sociological factors regard- 
ing the vfhole culture of the undereducated in relation to unenployment, 
under-anployment and welfare. Their adult students may find it diffi- 
cult to conmunicate. They may resent impersonal treatment, and they 
will certainly not want to be treated as children. Anxieties about 
family, financial matters, health, and jobs may nag them continually, 
making it difficult for them to concentrate on the facts and figures 
in their lessons. 

On the other hand, instructors need to learn to take advantage of 
the positive strengths that adults have, their wide ranging experiences, 
knowledge of life, ability to use reason, and their strong motivation 
to succeed lAen they realize the value of a goal and their own capability 
to achieve it. 

With understanding, the instructor can lead a student on to face his 
own mistakes and to do something about them, to make him realize that 
he is not doing work for the teacher but for himself. These are steps 
leading to responsibility for his own self -concept. In a program geared 
for his skill level, in lAich he is carefully placed and given work that 
he can do, he may begin to feel academically successful for the first 
time in his life. The effect will give him more self-respect. He will 
come to believe that he is capable of learning, and that he can move up 
a career ladder if he inproves his acadonic level. A feature of the LINC 
program in use is that many students, realizing success in learning, 
develop more ambition and higher expectations. Adult basic education 
gives them more than a certificate or entry into further training; it 
gives them self -confidence and a better control of their own lives. 

Both professionals and para-professionals can become excellent 
instructors for adults. Each has his own strengths. TTie pjira -professional 
may be more successful imnediately in establishing a good working rela- 
tionship with students; students may fear him le^s and be more willing 
to ask his help or to confide their problems to him as one ^o '•understands 
their language". On the other hand, a professional teacher is generally 
better able to diagnose learning difficulties and to prepare his own 
remedial vork sheets to help students overcome particular difficulties. 
If on top of his pedagogical skills he can also establish a synpathetic 
relationship with his students, the professional instructor can maintain 
his place as the best person for the job. 



D. LOCALIZING THE LINC PROGRAM 



Part of the training of instructors must be to help them to adapt 
the program to meet the standards and curriculum requirements of their 
own institution or province. 

Adaptation may be as simple as changing place names in word prob- 
lems and exercises in order to give a local flavour to the content. At 
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a slightly hi^er level, instructors may want to add a fev/ more objec- 
tives or items, or even to create a conpletely new unit. They can adapt 
tlie program gradually, starting with a fiill program on Day 1 of any 
training period and *(odifying it as time goes by or as needed change 
becomes apparent. In some cases, instixictors and curriculum development 
teams my plan to rewrite a complete course, using the LINC system as a 
foundation on which to build their new content. For them the LINC pro- 
gram becomes a model and a soiree of objectives and questions to borrow 
from. Some institutions ha^ - patterned other courses, such as science 
and conmercial subjects, on the LINC system. 

As part of their training, instructors are taught a model of curri- 
culum developnent v^ich follows six basic steps or phases. They can 
then use vdiatever they need from the model to localize the program. 



Phase 1 
Phase 2 
Phase 3 
Phase 4 
Phase 5 
Phase 6 



Define the terminal behaviors. 

Design an evaluation system. 

Define and cluster instructional objectives. 

Prepare evaluation instruments. 

.^enare learning activities. 

Prepare individual student activities. 



E. CONCLUSION 



The LINC program exein)lifies the principles and processes of 
individualization which many adult upgrading institutions in Canada are 
trying to implement. Use of the program encourages, and even forces, 
instructors to work at a one-to-one level with adult students. If in- 
structors use the program conscientiously, even after they expand or 
adapt it, they automatically put the principles and processes into ef- 
fect in the classroom. In this way the LINC program is a vehicle for 
innovation. 

LINC was designed as a core program with syncretic capabilities; 
it can absorb the best features of any program or materials already 
in use in a school. Its built-in system of placement, diagnosis, pres- 
cription, individual work with a variety of iraterials and methods, and 
evaluation provides a framework for varying content and standards needed 
to fit a local situation. 

Busy instructors may not have the time, ability, or inclination to 
develop a full program at once. However, by giving them a framework, a 
basic system complete and viable from the first day, they are relieved 
of much of the tedium of development, they have a model to follow, they 
can make changes and iinprovements gradually and they can be expected to 
conduct a very high level of relevant training in their classrooms. 



CHAPTER V 



COGNITIVE STYLES AND THEIR MAPPING FOR 
TRAINING AND DEVELOPMENT 



Bikkar S. Randhasf^a* 



Individual differences have provided a challenge both to res- 
earchers and practitioners. The xjnravelling of this challenge has 
alrea<fy occi^ied psychologists and educators for a nunber of years. 
The nystery of individual differences and the various dimensions of 
their variability has intrigued social scientists. Ihoug^ unequivocal 
resolution of the various dimensions of hunan variability is far from 
complete, several ccn^rehensive models £md theories have been proposed 
to illustrate the interactive effects of the various dimensions 
(e.g., Gardner, Jackson, and Missick, 1960; Guilford, 1967). 

It is evident that a few early workers in the area of mental 
measurement had anphasized differences in sensory and perceptual 
diaract eristics. Gal ton (1883) developed measures of sensitivity 
to tones and weight discrimination, and also atte^^)ted to deteimine 
the forms of imagery of subjects. Cattell and Farrand (1896) and 
Gilbert (1897) are also among the early researchers irfio measured 
perceptual differences . Even before the turn of the century 
accurate measurements of differences in perception were available. 
But the interest in such research declined. The reason seems to be 
that research in individual differences has a practical orientation 
and tends to be continued only when some potential applications are 
evident (Tyler, 1965). 

Costs of education, training, and developnent of materials have 
hit the all time high. Funding for these enterprises is coming under 
tough scrutiny and concern. The cost-benefit analyses do not present 
a bright picture for all concerned in our society. The disillusioment 
of educators, students, and administrators at the payoff of education 
is shared by the tasqpayers. The study of cognitive styles and mapping 
for educational training and development holds premise for increased 
returns from educational investments. 



^Professor, College of Education, University of Saskatchewan, 
Saskatoon 
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This paper is an attempt to review the ^elated lit^;rature in the 
area of cognitive (learning) styles, cognitive mapping, correlates of 
cognitive styles, effects of cognitive styles on learning, ciianges in 
cognitive maps and their concomitant effects. In a separate section 
of this paper important training and development implications will be 
discussed. 



Cognitive Style 



Framework . Individual differences among students in a given 
situation have long been known and studied by psychologists and ed- 
ucators. A number of personality and cognitive, or intellective, 
factors that might explain the individual variations in any group 
C- learners have been examined. Cognitive style is one of the most 
intriguing concepts studied in this area. However, implication cf 
a cognitive style ^preach to study of the classroom situation \s 
in its beginning stages. Accordingly, for some of the practical 
implications to be discussed in this report direct evidence is quite 
sparse . 

The concept of cognitive styles had its origin some twenty years 
ago in the observation of self-consistency in an individual's way of 
handling a wide range of perceptual and intellectual tasks. Hence, 
the designations "cognitive" and "style". Further research indicated 
that modes of functioning expressed in a given cognitive style could 
be found, in congruent foim, in the individual's personality and social 
behavior as well. Despite much evidence that cognitive style dimensions 
are clearly quite broad in scope, the original label persists. Because 
of the inclusion of extra-CQgnitive components involved in it, a des- 
ignation such as "personal styles" seems more appropriate (Witkin and 
Moore, 1974). Sigel and Coop (1974) certainly affiim the breadth in 
scope of this concept in stating that "cognitive style is an integrated 
concept in that its definition implicity bridges the personality - 
cognitive dimensions of the individual (p. 251)". Hill's (in an un- 
dated manuscript) us? of the concept of cognitive style is similarly 
broad in scope. He states that "an individual's cognitive style is 
deteimined by the way he takes note of his total surroundings how 
he seeks meaning, how he becomes infoimed (Hill, p. 3)". 

The concept of "learning style" certainly encompasses the broad- 
ened scope of the concept of cognitive style. For example. Nations 
(1967) perceived learning styles as a combination of sensory orientation, 
responsive mode, and thirling pattern. More details on learning styles 
will be provide! later. Also a brief discussion on the structure of 
intellect will be incoiporated in the section on learning styles. 

Distinctive Styles . Cognitive style refers to self- consistent 
and enduring individual differences in perceiving, ideating, or 
presenting aspects of the environment requiring symbolic mediation. 
The te'*n cognitive style refers, therefore, both to individual dif- 
feici. In general principles of cognitive organization (simplification 
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and consistency trends) , and to various self-consistent idiosyncratic 
tendencies (intolerance for ambiguity; memory for particular kinds 
of experience; preferences) that are not indicants of htman cognitive 
behaviours in general. It represents differences in personality 
organization as well as u genetically and environmentally deteimined 
variations in cognitive edacity and functioning. It is also seen 
as a mediator between motivation and emotion, on the one hand, and 
cognition, on the other (Paul, 1959). 

Ausubel (1968) listed several cognitive styles that liave been 
identified and studied. These are intolerance for ambiguity (tendency 
toward premature closure) ; intolerance for unrealistic experience; 
leveling - sharpening; need for simplification (skeletonizing, ration- 
alizing); degree of cognitive differentiati<Mi; explication and im- 
porting of detail in memory (embroidery) ; vividness of memory, long- 
term versus short- tern memory; memory for particular kinds and sense 
modalities of experience; rigidity or flexibility in problem solving; 
preference for cognitive complexity or simplicity, for widely known 
or little known infoimation; and preference for broad or narrow 
categorization. Other possible and suggested cognitive style constructs 
include strategy preferences in problan solving (scanning or focusing; 
the use of whole or part hypotheses) ; strategy preferei|ces in acquiring, 
organizir^g, and processing infoimation; memory for details or concepts; 
integration versus compartmentalization in memory; and degree of openness 
t^ new infoimation after closure is achieved. 

It must be emjAasized that the teim cognitive style has been 
used by a nunber of investigators to denote the different ways in 
lAich children and adults perceive and categorize their environment 
in specific situations in self-consistent and enduring manner. How- 
ever, each investigator has a different meaning in mind \A\en he defines 
his concept of cognitive style. The definition of cognitive style 
has mostly been dependent an the type of tasks that the researcher 
employed to elicit the responses on whidi the siibjects were classified. 
Therefore, there is no doubt that there has been some confusion re- 
garding the construct of cognitive style. It becomes essential, there- 
fore, to mention the investigator as a context in which a style is 
used. It is hoped that an examination of some of the most frequently 
used measures of cognitive style will help to clarify this confusion. 
Some of the major style categories along with exanples of the measuring 
tasks for categorization will be presented. 

Major Style Categories . The domain of cognitive styles is vast. 
This section will confine discussion to cognitive style categories 
that grew out of the leadership and direction of Broverman, Gardner, 
Kagan, and Witkin. 

a. Cognitive Style as Intraindividual Differences : Broveman 
(1960) conceptualizes cognitive styles as exi»ressions of different 
"response probabilities or response strengths in certain types of 
classes of behaviours (p. 167)". His conception of cognitive style 
is somewhat different from any of the other researchers in the area. 
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He views cognitive style as relationships between the fimctions within 
individuals . . . rather than . . . individual differences vdiich are based 
upon nonnative differences between individuals (Broverinan, p. 487)". 

It follows then that individuals have developed ways of responding 
to particular types of situations and that the chance of a particular 
response being used depends on the nature of the task and the intensity 
with whidi a particular response is held. fftid> of Broverman's recent 
researdi has been devoted to the delineation of two cognitive styles. 

The first cognitive style is conceptual versus perceptual motor 
dominance. This refers to "specialization in novel or difficult per- 
ceptual-motor bdiaviours, or vice versa (Broveiman, p. 487)". It has 
been demonstrated that a conceptual dominance is related to perfoimance 
on sectims of the Pr ^ry Mertal Abilities test, nonsense syllable 
fluency tests, Stroop word-Color test. Spelling Three Letter Anagrams 
test, and the Fifteen Clues test. However, perfoimance on card sorting 
tasks. Primary Mental Abilities Spatial subtest and toothpick construction 
test are correlates of a perceptual -motor dominance. 

The second cognitive style identified by Broveiman is a strcmg 
versus weak automatization. This is determined on the basis of 
performance m automatic habitual type of situations (siirple repetitive 
tasks) . Strong automatization is associated with the ability to com- 
plete single addition and subtraction problems, the ability to perfoim 
well on the Stroop Word-Color Interference test, and the ability to 
solve threee letter anagrams. However, weak automatization is cor- 
related with a better perfoimance on the Kohls Blocks, Porter's Maze, 
Goodenough Draw-A-Man test, Witkin's Bnbedded Figures test, Thurs tone- 
Go thsdialdt Figures test, and the Reasoning Subtest of the Primary 
Mental Abilities test. 

Broveiman's distinction between styles relative to task is 
rather interesting. It must be noted that other conceptual schemes 
of cognitive styles do not take the task into account. Broveiman 
proposes a functional relationship between style responses and types 
of task. It is very relevant and practical for educational appli- 
cations. 

b. Cognitive Control Principles; Cognitive controls are the 
result of woric of Gardner and his colleagues (Gardner, Holzman, 
Klein, Linton, and Spence, 1959; Gardner, Jackson, and Mes sick, 1960). 
Cognitive controls are the pervasive organizing principles that guide 
behavior towart a goal. Gardner and his associates perceive cognitive 
style from a psychoanalytic framework. Their notion of cognitive 
style incorporates a niinber of different principles all of vdiich 
function to assist the individual in adapting to a complex environment. 

Gardner and his co-workers have identified six control principles 
which in their totality describe an individual's style. These control 
principles are sharpening - leveling, equivalence range, scanning (focal 
attention) , tolerance for unrealistic ^erience, fldcible - constricted , 
and field articulation (field independence - field dependehceX! It is 
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interesting to observe that at the beginning these workers used the 
tern '•perceptual attitude" or its German equivalent Ansdiauung as a 
label for these personal ways of understanding reality"! However, 
it ASS realized that other cognitive processes besides perception 
were involved and cognitive control principles seemed to be a more 
accurate label. 

The sharpening - leveling dimension was the first of these control 
principles that was intensively studied. Sharpeners differ fron lev- 
elers in tlie degree of articulation of recall. When an individual 
experiences a series of similar stimuli, each new stimulus affects 
the memory of the preceding stimuli. The interaction of the perceptual 
processes and memory traces is called mutual assimilation. Sharpeners 
are less susceptible to the influence of mutual assimilation vriiereas 
levelers are relatively more susceptible to such influences. The 
Schematizing test developed by Hollingsworth is used most often to 
differentiate sharpeners and levelers. Tliis test involves the pres- 
entation of squares one at a time in random order in sets of five 
different sizes and the subject is required to judge the size of each. 
The levelers show significantly greater inaccuracy in their size 
judgpients than the sharpeners. The theoretical distinction between 
levelers and sharpeners is that levelers tend to assimilate new 
stimuli to an alrea^ dominant cognitive organization and thus not 
to be aware of differences between the new and the old, ^ereas 
sharpeners notice dianges and keep successive stimulating situations 
distinct from one another. 

The second cognitive control variable is equivalence range . This 
is usually measured by an object-sorting test in which a subject is 
free to use as many or as few categories as he wishes. Individual 
differences in this variable are self -consistent and endurable. 

Another cognitive control principle is scanning (focusing) . 
The test used to measure it is usally a size estimation task of some 
sort and the siibject is required to match a variable stimulus to a 
standard. Individuals who focus on the standard tend to overestijnate 
its size and those vtfio scan the vtfiole situation more broadly show 
much less overestimation. 

The fourth cognitive control principle is referred to as tolerance 
for unrealis tic experience . Tolerance for unrealistic experience 
"involves individual differences in willingness to experience, or 
to produce, organizations of stimuli at variance with what tne subject 
'knows' to be true in a relatively literal and conventional sense 
(Gardner and Moriarty, 1968, p. 50)". It can be measured by an apparent 
movement test in vAidi the alternation rate of the stimuli can be 
controlled. The score used to measure this principle is the range of 
alternation rates over vAiidx the illusion of movement occurs. 

The fifth cognitive control principle is called flexible versus 
constricted control . Much less supplementary evidence is available 
on this-variable than the previous ones. It is measured by the Color- 
Word test, in vtfiich after reading the names of colors, and naming 
actual colors presented in stimulus strips, the individual is pre- 
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sented with color names painted in conflicting colors and asked to 
nane each color, ignoring the word. In the study of individuals , four 
consistencies referred to ^cve this co itrol variable did nov seem 
to define a separate factor (Gardner, et al, 1959). Therefore, the 
status of this control principle as a separate variable is a bit 
doii>tful. 

The sixth control principle anerged fron the evidence of separate 
existence of Wilkin's field independence - dependence dimension. 
However, Gardner and his colleagues preferred! to call it field artic- 
ulation . This principle will be discussed later in the section on 
field independence -dependence style. 

c. Kagan's Stylistic Modes: Kagan, Moss, and Sigel (1963) 
define cognitive style as a ^'teim that refers to stable individual 
preferences in mode of perceptual organization and conceptual cat- 
egorization of the external environment (p. 74)". They describe 
analytic - descriptive , inferential-categorical , and rel ational -contex" 
tual tendencies among siibjects in gnxying pictures. These tendencies 
of perceiving and categorizing are based on individuals* responses 
to pictorial stimuli. Conceptual Style Test. 

An analytic - descriptive response reflects the tendency to 
classify Items on the basis of objective, observable diaracteristics 
that are part of the total stimulus situation. This response mpde 
is typically characterized by a splitting of the stimulus situation 
into conqponent parts and attending to these discrete pieces as a 
basis for categorizing the total stimulus. 

The inferential - categorical response mode (style) is exanplified 
by an individuafTSw chooses to toim his categories on the basis of 
inferences made about the pictures he groiq)s together. No unitar/ 
objective diaracteristic is partialled out by the individual as the 
basis of classification. 

• A relational - contextual style is manifest v4ien pictorial items 
are grouped together on the basis of tenq)oral, functional, or spatial 
contiguity. It has been observed that several times these responses 
reflect a certain theme or story line that integrates the various 
items characterized in a relational manner. 

As Kagan pursued Ids woik with the three dimensions of cognitive 
style discussed above, he observed that many subjects tended to take 
a long time to reflect over various stimulus alternatives before making 
a response, and other individuals tended to give a very quick response 
to the tasks presented them. Kagan was impressed by the difference 
in response uncertainty (i.e., two or more highly plausible response 
alternatives). He developed a test called the Matching Familiar Figures 
Test (MFF) to determine response ti-ne and errors made by the subjects. 
Based on this test Kagan proposed a new cognitive style called con- 
ceptual tempo . 
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Conceptual tempo reflects the speed with which an individual 
responds to tasks of high response uncertainty and the ranter of 
errors made in the responses. Those siijects who respond fast and 
make many errors are called impulsive . However, those respcmdees 
v*o ere slower in responding and make fewer errors are called 
reflective . It has been observed that generally subjects with 
analytic - descriptive style of responses on the Conceptual Style 
Test tend to be reflective. 

d. Field - Dependence - Independence Cognitive Style : Field - 
dependence - independence dimension ot cognitive style is based on 
a global -analy tic- continuun . "Relatively field- independent persons 
tend to experience parts of the field as discrete from the surrounding 
field, even when the field is so organized as to strongly embed the 
part; that is, they perceive analytically. Perception of relatively 
field-dependent persons, on the other hand, is guided by the organ- 
ization of the field as a v*ole, so that any part of the field is 
ejqwrienced as continuous with its surrounding; that is, their per- 
ception is global (Witkin and Moore, 1974, p. 2)". Witkin's Bnbedded 
Figures Test is used to identify field- independent and -dependent 
subjects. This test atteaipts to detemine the extent to which in- 
dividuals are able to overcome the effects of distracting bad^round 
elements (the field) vAen they are attempting to differentiate re- 
levant aspects of a particular situation. This cogrdtive style 
concept is very thoroughly discussed in Personality Throu gh Perception 

(Witkin, et al, 1954) and Psycholoai.cal Differentiation (Witkin, 

et al, 1962). There is perhaps mc.e research evidence on this style 
than any other cognitive style identified thus far. However, very 
active research work is being pursued in conceptual tempo and evidence 
on this style is very rapidly accumulating. It must be noted that 
analytic style described by Kagan, et al (1963) is quite different 
from the description given this tern by Witkin et al (1954) . Bnpirical 
evidence substantiates the distinctiveness of the two styles described 
by these c/o schools of thought (Coop and Hovenden, 1967; Frehner. 
1971) . 

Rev iew of Research . Many interesting relationships have emerged 
from research involving the various notions of cognitive style. There 
is not enough research evidence that bears directly on practical ed- 
ucational problems such as match -mismatch of cognitive styles in 
learners and instructors and its consequent interactive effects on 
dependent vai iables . Only a start has been made in this direction 
and much more needs to be done. Witkin and Moore (1974) point out 
a number of fruitful research avenues that can be pursued to tackle 
some of the crucial ed'icational issues. 

Review of research in this section will be limited to only those 
relationships that have educational (curriculim and training) impli- 
cations. However, for a review of the many results that deal with 
theoretical and methodological matteis, see Kagan and Kogan (1970) . 
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Witkin and Moore (1974) argue that "field dependent" and "field 
independent" are labels for clusters of characteristics, both cognitive 
and personal, justifying the designation "personal styles". Two com- 
prehensive reviews on Witkin's notion of personal styles and their 
relationships with educationally ^^levant variables are very significant 
sources for researchers (Witk' Witkin and Moore, 1974). Only 

conclusions from these two S' Vf^il be provided in this section. 

"It is now well documented that relatively field-dependent persons 
are drawn to people, both in the sense of being interested in what 
others say, do and feel, and in the sense of liking to have people 
around them (Witkin and Moore, 1974, p. 3)". The field-dependent 
people have been found to favor occipations vAich require involvement 
with others -- for example, selling, elementary school teaching, 
rehabilitation counselling. Preferences of field- independent persons 
are found to be for those occiqpations in which working with others 
is not especially important -- for example, astronomy, engineering. 
Field-dependent persons prefer academic subjects in le social 
sciences when in school whereas field-independent persons tend to 
favor mathematics and the physical sciences (Witkin, 1973) . 

Field-dependent persons have been found to be particularly 
responsive to nuances of emotional expression in people around them. 
Also these persons tend to take greater account of external social 
referents in defining their own points of view and feelings (Witkin 
and Moore, 1974). Wiat are the implications of these distinctive 
social -orientations of field-dependent and field- independent persons 
for how they learn and what they leam? 

"Field-dependent and field- independent persons are not different 
in sheer learning ability or memory. However, reflecting differences 
between them in interests, and hence in what is attended to, salient 
and relevant, field-dependents are better at learning and remembering 
social material and field- independents are better at learning and 
rem«nbering impersonal material (Witkin and Moore, 1974, p. 6)". It 
can be concluded from the above discussion that field-dependents and 
field- independents cannot be arrayed along a better-worse continuum 
on learning tasks. Each could be expected to do better at tasks 
requiring matching basic orientation. 

Witkin and Moore (1974) have, on the basis of a sizeable literature 
stated the effect of social reinforcement, "one of the handiest tools 
in the teacher's armamentariun of devices for perpetuating some student 
behaviours and obliterating others," on persons with field-dependent 
and field- independent orientations. They conclude that social rein- 
forcement has a significantly decisive effect on field-dependents 
and field- independents. However, the effect of response -contingent 
reinforcement is differential on persons with these two orientations. 
The results of studies of Ferrell (1971) and Fitz (1970) indicate, 
by and large, that negative reinforcement, administered immediately 
after a response is made, has greater effect, in the form of better 
learning, on field-dependent than field- independent persons. Positive 
reinforcement following a response seems to have littlj effect on 
either kind of learner. 
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The organizational nature of the learning material again has 
differential effect on field-dependent and field- independent learners. 
Whatever the nature of the material field-dependent learners are 
required to learn, they are more likely to take its organization as 
given, rather than attempt to impose an organization of their own, 
Witkin and Moore (1974) further state that often in learning, the 
material to be learned lacks clear inherent structure, creating 
thereby the need that learners tharaiselves provide organization as 
an aid to learning. Fie Id- dependent learners are found to have more 
difficulty in learning such material as compared with field- independent 
learners. For the well organized material requiring no further struc- 
turing, field- independent and field-dependent learners are not likely 
to differ in their learning of it (Bruce, 1965), 

Since field- independents are better able to organize their ex- 
perience it is postulated that they may be better able to provide 
their own strategies for coding and utilizing infoimation and are 
less reliant on strategies that may be provided by the task or by 
someone else, Koran, Snow, and McDonald (1971) investigated indiv- 
idual differences in the acquisition of a teaching skill from written 
and video-modeling procedures. They found that these two treatments 
were differentially effective for learners of different orientations. 
Video -modeling was more effective in general but field-dependent 
teachers benefited equally or more fron the written modeling. 

For training, instruction, or therapy, the question to be 
answered is this: What are the effects of an interaction when its 
participants are matched or mismatched on cognitive style? DiStefano 
(1969) investigated interpersonal perceptions of teachers and 
students in regular classroom situations. He found that teachers 
and students matched to each other in style perceived one another 
positively, whereas students and teachers who were mismatched per- 
ceived each other negatively. It is interesting to note that the 
positive and negative evaluations included not on / personal char- - 
acteristics but intellectual characteristics as well. In a recent 
study, James (1973), using a specially created mini-course confirmed 
DiStefano 's results. Similar results in feelings of patients 
toward their therapists in matched and mismatched dyads were obtained 
by Folman (1973) and Greene (1972), 

In summary, research involving field-independent and field- 
dependent styles has indicated that field- independent persons tend 
to be analytic in intellectual approach, sanewhat distant from others, 
high achievers, emotionally independent, and active and controlling 
their environments rather than being controlled by them. Field- 
dependent persons are found to be the opposite of fie Id- independent 
persons in the variables listed above. 
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The variables summarized above in connection with Witkin's notion 
of cogniti\e style have also been studied by Kagan and others (Kagan, 
Moss, and Sigel, 1963; Kagan, Rosman, Day, Albert, and Phillips, 1964; 
Sigel, 1963; Broverman, 1960). Kagan, et al descriptive-analytic 
styles are also related to personality, social, and intellectual char- 
acteristics. Most of the reported research for these styles involved 
children. Only a few studies are laiown to have used adults. 

Kagan, et al (1963, 1964) found sex differences on analytic- 
descriptive and relational -contextual styles. Boys employed more 
analytic-descriptive whereas girls employed more relational- contextual 
responses. Analytic-descriptive responses were related to nonverbal 
intelligence tests, learning of concepts, and memory for details. 
Also analytic responses of males were related to cautiousness, learning 
skills, achievement orientation, independence and activity. The reverse 
held true for girls on most variables. 

Kagan and Kogan (1970) reported that reflective- impulsive style 
was consistent across situations. It was also reported that reflec- 
tives made fewer errors than impulsives in recall, reading, and 
reasoning. Persons high on inferential-categorical responses were 
found to be also high on the Binet intelligence test. These results 
were interpreted as an indication that inferential- categorical style 
may well be a product of conventional socialization experiences 
(Sigel, Jarman, and Hanesian, 1967). 

Serafica and Sigel (1970) found that inferential -categorical 
style identified good and poor readers. The readers tended to yield 
more inferential -categorical responses than did the nonreaders. The 
authors interpreted this result as suggestive of the importance of 
the role of interpretive -conceptual processes in reading, as compared 
to the emphasis on discrimination of particular details. 



Cognitive M^s 



E.C. Tolman published a paper in Psychological Review (1948) 
entitled "Cognitive Maps in Rats and Men''. Tolman presented in 
this paper a description of various learning experiments with rats 
and suggested in the conclusion of it the implications of his rat 
work for the purposive behaviour of men. He admitted his affiliation 
with field theorists and asserted that 'Ve believe that in the course 
of learning something like a field map of tbft env^.ronment gets 
established in the rat's brain (Tolman, 1948, p. 192)". The theo- 
retical position Tolman took in the above paper contained two 
assumptions. "First, that learning consists not in stimulus -response 
connections but in t!ie building up in the nervous system of sets 
which function like cognitive maps and second, that such cognitive 
maps may be usefully characterized as varying from a narrow strip 
variety to a broader comprehensive variety (p. 193)". 



ERIC ixl 



107 



Tolman (1932, 1959) listed four main kinds of individual dif- 
ference variables vdiich are relevant for the purposes of this paper. 
The four are heiedity, age, training, and endocrine, drug, or vitamin 
conditions. Using the initial letters of these four, ToLnan called 
them the HATE variables. As can be noted, only one of the HATE 
variables involves learning. Individual differences througji learning 
(training) could also be inferred from the cognitive maps of learners 
(from the broad con5)rehensive to the narrow strip map continuum). 
However, Tolman (1948) suggested that "narrow strip maps rather than 
broad comprehensive maps seem to be induced: (1) by a damaged brain, 
(2) by an inadequate array of environmentally presented cues, (3) by 
an overdose of repetitions on the original trained-on path, and 
(4) by the presence of too strongly motivational or of too strongly 
fnistrating conditions (pp. 205-207)". These conditions should be 
of paramount value vdien taken into account in child rearing and dir- 
ectii;g the learning of individuals in institutions or clinical settings. 

Now operationally defined a cognitive map is a picture of the 
diverse ways in vdiich an organism acquires meaning. It identifies 
his cognitive strengths and weaknesses . This infoiraation can be used 
to build an indivi(hialized and personalized program of instruction 
or training (Hill, undated manuscript). Hill provides an illustration 
of a personalized program of instruction through cognitive mapping 
in a series of publications under the general heading "The Educational 
Sciences". 

It must be emphasized that a cognitive map is simply an inferred 
description of the cognitive traits of an individual. Though theo- 
retically we expect an infinite number of distinctive cognitive traits 
in the population at large, yet grouping and reducing the dimensions 
of vari*ility in cognitive patterns a manageable number of maps for 
instmctional prescription can be obtained. However, research with 
match-mismatch of learner cognitive map with the intended instructional 
strategy is only its infancy. Once again we are in a situatioii where 
theory is far ahead of its implementation and the associated research. 
A learning style is essentially a cognitive map of the learner. In 
other words, Witkin's (1974) suggestion to refer to cognitive styles 
as personal styles is in essence broadening the scope of cognitive 
styles as we have known than till now. Thus the broadened scope of 
cognitive styles would more appropriately be referred to as learning 
styles. 



The Structure of Intellect 
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A morphological model of intelligence was first proposed by 
Guilford in 1956. In astronomy, similar analysis was applied by 
Zwicky (1957). Guilford^s (1956, 159, 1967) model of intelligence 
attenpts to cross-classify abilities in three different ways, and 
the categories of one dimension of ability intersect with abilities 
of the other two dimensions. A graphic representation of the structure 
of intellect model is given in Figure 1. 
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Figure 1. STRUCTURE OF lOTELLECT OJBE 
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One way of classification is in teims of the kind of mental operation 
involved in the abilities. An operation is a major intellectual activity 
or process; things the organism does with the raw materials of infoimation. 
Infomation in this context is "that which the organism discriminates'*. 
Eadi ability involves simply one of the five operations outlined in 
Figure 1. Each operation category is depicted as encompassing 24 
different abilities which are parallel to those in every other operation 
category. 

The second way of classification is in teims of content . Contents 
are broad, substantive, basic kinds or areas of infomation within 
which the operations are perfomed. Each of the four content categories 
includes 30 abilities that are parallel to those in every other content 
category (see Figure 1) . 

The third dimension of classification of abilities is in teims of 
roduct . "Whereas we may say that the content categories describe the 
asic substantive kinds of infoimation from the psychological point of 
view, the product categories describe the fomal kinds of infoimation 
(Guilford and Hoepfher, 1971, p. 18)". There are six product categories 
as shown in Figure 1. Each product category contains a set of 20 
abilities vrfiich are parallel to those in every other product category. 
Products are "basic forms that infoimation takes in the organism's 
processing of it (Guilford and iloepfner, 1971, p. 21)". 

All three classifications (operations - 5, contents - 4, and 
products - 6) put together in one cross-classification produces the 
model illustrated in Figure 1. There are a total of 120 little cubes 
(5 X 4 X 6) or cells in the model. Each little cell in the model 
represents theoretically a unique kind of ability by virtue of its 
unique combination of one kind of operation, one kind of content, and 
one kind of product. For example, it may be the cognition of figural 
units (CFU) , the memory of symbolic relations (MSR) , or the evaluation 
of behavioural classes (EBC) . 

It must not be assumed that 120 abilities cover the entire range 
of intellective traits or variables. With thorough investigations 
there are reasons to expect more than the nunber postulated in the 
structure of intellect. Evidence has been found that at least three 
of the cells already have two abilities represented in each of them 
(Guilford and Hoepfner, 1971). The cognition of figural units (CFU), 
the cognition of symbolic units (CSU), and the memory for figural 
systems (MFS) cells have visual and auditory abilities. The cell for 
cognition of figural systems has not only a visual and an auditory 
ability but also appears to have a kinesthetic ability. It it likely 
that within the cognition and manory operation categories, at least, 
there may be quite generally differentiation of abilities along sense- 
modalities. This may also prove to be the case in other operation 
categories for figural and symbolic categories of content. Future 
research in these areas may determine the existence of sense modalities 
as a new dimension for a complete model of intelligence. 
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It must not also be assuned that, "although the abilities are 
separate aiid distinct logically and they can be segregated by factor 
analysis, they function in isolation in the mental activities of the 
individual. TWo or more abilities are involved in solving the same 
problem (Guilford and Hoepfiier, 1971, p. 19)". 

Major Dimensions afid Subcategories . Major dimensions and their 
categories are defined below. Examples of abilities representing the 
categories along with other elaborations are also given. The codes 
used to represent these categories are given in the parentheses 
following the category name. These codes are also evident in Figure 1 
from the capital letters used in the designations of the categories. 
Each structure of intellect ability is generally designated in teims 
of its special trigram which is composed of a letter from eadi of its 
dimensions its operation, content, and product, in that order. 
For example, CSU is an abbreviation for "cognition of symbolic units" 
and B^S represents "evaluation of semantic systems". A unique definition 
can be obtained for each cell, ability, by simply specifying its char- 
acteristics in teims of the represented categories from the three major 
dimensions . 

erations . Major kinds of intellectual activities or processes; 
ings that the organism does in the processing of infomation, 
information being defined as "that which the organism discriminates". 

Cognition (C) : Immediate discovery, awareness, rediscovery, 
or recognition of information in its various forms; compre- 
hension or understanding. 

Monory (M) : Fixation of newly gained information in storage, 
ihe operation of manory is to be distinguished from the 
memory store. 

Divergent production (D) ; Generation of logical alternatives 
from given infonnation, where the emphasis is upon variety, 
quantity, and relevance of output from the same source. 
Likely to involve transfer recall (instigated by new cues) . 

Convergent production (N): Generation of logical conclusions 
from given infomation, where emiiiasis is upon achieving 
unique or conventionally best outcomes. It is likely that 
the given (cue) information fully determines the outcome, 
as in mathematics and logic. 

Evaluation (E) : Comparison of items of information in terms 
of variables' and making judgjnents concerning criterion sat- 
isfaction (correctness, identity, consistency, etc.). 

Contents . Broad, substantive, basic kinds or aieas of infomation. 

Figural (F): Pertaining to information in concrete fom, 
as perceived or as recalled in the fom of images. The 
tern "figural" minimally implies figure-ground perceptual 
organization. Different sense modalities may be involved 
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visual, auditory, kinesthetic, and perhaps others. 

Symbolic (S): Pertaining to information in the form of 
denotative signs having no significance in and of them- 
selves, such as letters, nuiibers, musical notations^ 
codes , and words (as ordered letter cGmbinations) . 

Semantic (M) : Pertaining to inf oimation in the foim of 
conceptiors or mental constructs to ^^ch words are often 
applied, hence most notable in verbal thinking and verbal 
commjnication, but not necessarily dependent upon words. 
Meaningful pictures also convey semantic information. 

Behavioural (B) : Pertaining to infoimation, essentially 
nonfigural and nonveifcal, involved in hunan interactions, 
vdiere the attitudes, needs, desires, moods, intentions, 
perceptions, thoughts, etc. of others and of ourselves are 
involved. 

Products . Basic forms that infoimation takes in the organism's 
processing of it. 

Units (U) : Relatively segregated or circunscribed items 
or "chunks" of infoimation naving "thing" character. May 
be close to Gestalt psychology's "figure on a ground". 

Classes (C): Conceptions underlying sets of items of 
infoimation groi^d by virtue of their ccnmon properties . 

Relations (R) : Connections between items of infoimation 
based i^n variables or points of contact that apply to 
them. Relational connections are more defin^* \e than 
implicational connections. 

Systems (S) : Organized or structured aggregates of items 
of infoimation; complexes of interrelated or interacting 
parts. 

Transformations (T) : Changes of various kinds (redef- 
initions, shifts, transitions, or modifications) in 
existing information. 

Implications (I) : Circunstantial connections between 
items of infoimation, as by virtue of contiguity, or 
any condition that promotes "belongingness". 

Learning Styles and Multiple-Abilit/ Conception of Intelligence. 
It has been pointed out earlier tiiat learning styles are the cc^nitive 
maps the various w^s learners derive meaning from tlie enviroiment. 
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Tolman (1948) asserted that in the course of learning something like 
a field map of the environment gets established. In other words, learning 
produces intellectual development of various kinds. The structure of 
intellect (SI) model provides a basis of deteimining how an individual 
acquires meaning. This model holds potential for assessing 120 dis- 
tinctive abilities, though all of the 120 SI abilities are not demon- 
strated as yet. About 100 SI abilities that have already been demon- 
strated offer an optimistic note fo^^ all those interested in individual 
differences. We can no longer be satisfied with the notion that intellig- 
ence is a simplicity liiich can be singularly expressed quantitatively. 

Ihe SI model and the research associated with the explication of 
it provide not only a means to express intelligence as multiple abilities 
but also provide the means to train and foster intellectual development. 
The SI profiles offer parameters for material to train abilities. They 
also offer a basis for individualized teaching Oteeker, 1969). It would 
ai^ar that the SI model may be considered as a con^rehensive medianism 
in vAiidi the learning styles are embedded. 

Researdi and Practical Issues . Guilford (1967) and Guilford and 
Hoepfher (1971) provide theoretical and technical details on the SI 
abilities. The latter work reported that 98 of the 120 SI abilities 
had been demonstrated. These abilities represented by the trigrams 
are presented in Table 1. It is clear from this table that all of 
the abilities (cells) in the behavioral content except cognition and 
divergent production have yet not been demonstrated. Meeker (1969) 
reported that about 79 abilities were demonstrated. She provides an 
excellent coverage of about 79 abilities with practical suggestions 
for curriculum, instruction, and remediation of deficits in these 
abilities. 

It is interesting to note that Guilford and his associates adnit 
the effect of training on intelligence. Another jolt is thereby 
administered to the geneticist position. Guilford (1967) stated that 
"if we grant that one of the major objectives of education is to foster 
intellectual growth of individuals, the new information regarding the 
nature of intelligence should not only contribute to the jMlosophy 
of eciication by helping to refine that objective but also provide 
suggestions for implementing it. It is now better known what intel- 
lectual development means (p. 475)". 

Myers and Torrance (1964) have designed and published formal 
exercises suitable for training sdiool children in intellectual skills. 
Materials for college students was developed and used by Upton and 
Samson (1963) and Fames (1961) developed materials for adults in general 
for training in intellectual skills. 

Teadiing methods and techniques as well as examinations can be 
significantly guided on the basis of the SI model. The way in which 
a subject is presented may make all the difference in the world as 
to whidi abilities are enq)hasized. The nature of the examinations 
used largely determine the kind of preparation students do in order 
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to do well on then. Specific techniques of presentation of material 
to be learned and the kind of exaninations to be used to detezmine 
the extent of that learning can be suggested by the nature of the SI 
abilities and the tests designed to measure then. 

Another practical implication of the SI model is "that each person 
vof be rather uneven in his many aspects of intelligence (Guilford, 
1958, p. 28)". Ihe common measures of ability often conceal such 
strengths and weaknesses in intellectual functioning. A person can 
often cover a weakness by having other strengths. However, some weak- 
nesses m^ not be easily cooqpensa'ed. Thus it is not uncanxm to find 
intelligent non-readers, non-speliers, non-writers,, etc. The SI abilities 
could provide a clearer diagnosis of particular stroigths and weaknesses 
in terms of the relevant factors. 

Inqilications of the SI model should be obvious for educational 
and vocational guidance and counselling. When advising a client to 
pursue the direction of his strraigths and to avoid the direction of 
his weaknesses, those stronger and weaker aspects must be known as 
clearly as possible. Much more analytic information than is jnrovided 
by an IQ or even a half-dozen different scores is required. The 
multiple-ability conception of intelligence provides the user with 
this potential. 
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Table 1 

Structure Of Intellect Factors That Have Been Deimmstrated 



Operation 
Qitegories 



Figural 



Cwitent Categories 
Symbolic Semantic Behavioral 



Number Khoivn 



Cognition 



Memory 



Divergent 
production 



Convergent 
production 



Evaluation 



Nmber known 



CFU 


CSU 


OH 


CBU 


CrC 


CSC 


CMC 


CBC 


CFR 


CSR 


(Ml 


CBR 


CFS 


CSS 


OS 


CBS 


CFT 


CST 




CBT 


CFI 


CSI 


CMI 


CBI 


MFU 


MSU 






MFC 


MSC 






MFR 


MSR 


hMR 




MFS 


MSS 




- 


MFT 


MST 






MFI 


MSI 


KMX 


• 


DFU 


DSD 




DBU 


DFC 


DSC 


IMC 


DBC 




DSR 


tMR 


DBR 


DFS 


DSS 


IMS 


DBS 


DFi 


DBT 




DBT 


DFI 


061 


IMI 


DBI 






Mil 




?.VC 


NSC 


NNIC 




NFR 


NSR 


NMR 






NSS 


NN6 




NFT 


NST 


NMT 




NFI 


NSI 


NMI 




EFU 


ESU 


EMU 




EFC 


I3SC 


EMC 




EFR 


ESR 


EMR 




EFS 


ESS 


ENS 




EFT 


EST 


EMT 




EH 


ESI 


ENa 




27 


29 


30 


12 



24 



18 



23 



15 



18 



98 
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CHAPTER VI 



OOURSE DEVELOPMENT FROM 
THE DEVHLOPER'S VIEWPOINT 



Naida Waite* 



A BTSD program may be open in the sense that anyone is eligible 
to enroll. This inplies that adults will enter with many different 
backgrounds of experience, and many different levels and conbinations 
of academic adiievement, and that their instruction will need to be 
individualized. 

The program nay be open in the sense that the learning will not 
be done in a formal classroom on a set schedule. In order to avoid 
over-dependence on reading and to achieve certain objectives, it may 
be advisable to introduce oral materials, pictures (still or navina) . 
manipulative materials, or other media, in various confcinations. 

In this w^ the student will have the advantage of spoken instruc- 
tions, explanations, practice sequences, drills, exanples, models, 
oral tests, pictorial illustrations, motion-picture demonstrations of 
processes, and so on, at the time vben he needs them. He can repeat 
these dynamic sequences until his learning needs have been met. 

Within the general topic of open training programs, this paper 
will deal with the process of developing a conplete individualized 
bodS"*^ includes the use of media in addition to printed 

Fbllowing is a skeleton siainary of the total process, after which 
there will be an elaboration of each step, sometimes in the fom of 
questions. You might consider it as a course development check list. 



* Educational Consultant and Program Developer 
Training Research and Development Station 
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CDURSE EEYELOPheJT OffiOC LIST 



list terns 
of 

reference 




List learner 
characteristics 
and needs 




Investigate 
instruction 
alrea^ 
available 


If it is 

suitable. STOP HERE 




possible, consider other \ 
priorities 


: unusable, 00 AHEAD ^ 



List content 

(1) Tentative 

(2) Necessary 
in form of 
critical objec- 
tives with 
"topics" under ^ 
eadi 



1 L 

j 

Consider strategies: 


Search out 


Instructional 


relevant 


approach 




Student* s dynamics 




Media 




Evaluation procedure 




(Budget) 




etc. 





Decide on 
strategies 

-T^m U 

List special 
services and 
equipment 
required 



Write a rationale 
of the course 



8 

Arrange the 
content in 
pedagogical 
order 



Divide coTitent 
into lessons 
or units 




10 

Write tentative 
objectives for 
Ikiit 1, convert- 
ing "topics" to 
sih-dbjectives 



11 

Write instruc- 
tional material 
for Iftiit 1 




12 

Re-examine the 
Udit 1 draft 
after an 
interval 
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13iA) 

Draft the n-^xt few units 
in the same way 



13CE) 



Insure availability of 
special staff, services or 
equipment to be reqtdred 
at a stated time 
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14CA2 

Have a few mits produced for 
developmental evaluation 



rr 



mi 




Axiange for a few represen* 
tative students to try the 
course vfcere you can iDonitor 
the process and results 



15(A) 

Gcnduct the 

developnsntal 

evaluation 




Keep a list of 
proposed dianges 
ndnor * major 



16 

Make any 

major 

revisicns 




17 

Re--evaluat3 
units with 
major 
changes 



18 



Recycle as \^ 
many times as \ 
necessaiy 


Conplete the development of 
the course - remaining uiits , 
general items ^ etc. 






19CB) 






Plan for the field test 




4 


) 19 CO 





20 



Have field test 
edition produced 
and distributed 



21 




Conduct 

field 

test 



Gradually substitute actual 
art work, etc. for any 
makeshifts used during 
development, as these proper 
items are received 



22 



Analyze feedback 
froa field test 




23(A) 



Revise as necessaiy or desired, 
and evaluate any major dianges 

— fr 



24 



2 3(B) 




Consider packaging of Itevised 
Edition 



I 



23(0 



Consider volume to be produced 



Have revised 
edition produced 
and disseminated 
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List Teims of Reference 

What is it you want to acconplish? 

Fbr vAiom? 

Who is sponsoring the development of the course? 

(It is not practical to attempt developmmt of a long course 
without a sponsor.) 

What ars the sponsor's areas of interest or responsibility? 

Is there a stated deadline , data, or is a deadline inplied by the 
circumstances? 

(Deadlines are something to be cautious about.) 

What funds will be available? 

What professional help? 

What supporting assistance? 

(as to typing, printing, art work, photogr^^ihy, acting, 
voice recording, editing, t^ duplicating, equipment, 
materials.) 

List Qiaracteristics and Needs of Learners 

Possible types of learners as you think they are: 

What do they need , and ^at do they want? 

(When the answers are different, peihaps you can present 
the needs in terms of the waits.) 

Academic badcground 

Socio-eccnondc level 

(What can they afford, if the course is not free to tibm?) 

Sociological characteristics 

(What would i^al to most? What mi^t offend somB?) 

Nunbers involved 

Geographic ;Location(s) 

Availability for training 

Investigate Instruction Already Available 

Is there suitable instruction? 

(No course will cover exactly vdiat you want, in exactly the 
wiQT you would dioose to teach it. Perhaps two sets of materials 
could be conbined, and there my be several q>proaches that 
would woik.) 



123 



Is there instruction that could possibly be used, though not entirely 
satisfactory? 

(Other priorities ndg^t make it inadvisable to lauidi into 
the long process of developing a course.) 

Is there nothing usable? 
(Ihen go ahead.) 

4. List Content 
Tentative content: 

Describe the critical objectives (main points) 

List possible topics under each 

(Consult esdsting courses, occupational supervisors, 
generic skills reports, etc. Behavioural objectives 
take tine to write, and you may delete sone of the 
content, so it is usually not efficient to write the 
detailed sti)-objectives at this stage. However, you 
almost have to write the critical objectives in order 
to clarify vihat you hope to do.) 

Necessary content: 

Delete all but the content that may be needed by the learners 
you have described. 

5. Consider Strategies 

(1) Strategies include 

The broad instructional e^roaches: 

(e.g., new or old math, a specific method for reading 
instruction, separate subjects or integrated 
materials, spiral or linear curriculun.) 

Ihe student's environment 
At home? 

In a conmnity groip? 

In a library or other informal learning centre? 

Ihe student's contacts 

By mail only? 

Face to face 

with a peer groip? 

with a tutor in a learning centre? 

at the student's discretion? 

at the tutor's discretion? 

both? 
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The student's activities (^mandcs) 

When he is leiaining» proving he has learned, or applying 
nd'.at he has learned: 

Will responses be by 

Mapoipulation? 

Mental response? 

Oral an tape? 

Oral to himself? 

Oral to a peer learner? 

Written? 

Written in a book? 

(If so, the book will serve only one student.) 

Drills on facts? 

Oral stimulus and response? 
Using stu^ cards? 

Any peer interaction? 

Self- tests intedded and/or separate? 

An instructor involved? If so. 
What inputs fron instructor? 
Initiated by %dun? 

What other diedcs or guidance to nonitor student's. 
progress? 

Will a certificate be issued on oaqpletion? 

(If so, the course oust provide criteria for 
certification; Tequiiements of orgaiizaticns 
demanding certificates must be considered.) 

Media: 

What media would best provide the desired dynamics (think 
big at this point)? 

Which of these media have you the resources to produce, in 
terms of 

services, equipment and materials available? 
time involved? 
budgetary considerations? 

As to the users, «Mch of these media 
can they afford to buy^ 

have they the capability of using? (e.g., is power 

available?) 
are purdiasable with dependable lead tine? 
can DO serviced in the area'r 
are not sii)ject to frequent breakdowns? 
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In lis^t of the above, whidi madia are practical to 
consider? 

Of these media, could cne or more be eliminated and the 
fuiction(s) performed by some of the others? 

(Ycu dcn't need to strive for a large variety; the 
fewer, sinpler, more convenient and less expensive 
the media are, the more use will be made of the 
course - provided the media you select will do the 
job required of them J 

Fbrmats: 

What formats will 

best present the content? 

appeal to students? 

(colour printing is attractive but e3q[)ensive; 
use of coloured paper is sometimes a good 
substitute 0 
facilitate storage and retrieval 

(use of colour may be helpful; control codes.) 

facilitate control of stocks 

(control codes) 
facilitate multiple use 

(many booklets, short tapes; all materials 

reusable.) 

If books are not to be written in, what will the student 
retain f6r reference when he finishes the course? 

Nothing? 

A sumaxy of some sort? 

An orderly set of his own woric, organized with this 
end in view? 

Wiat formats have you the resources to produce, in terms of 
services, equipment and materials available? 
time involved? 
budgetary considerations? 
Evaluation: 

The bases for evaluation need to be considered before the 
course is constructed, and may be built in to a large 
extent: 

How can the critical objectives (Step 4) be tested? 
(You may wish to rewrite them in more specific 
fbxm * e.g., the degree of proficiency required.) 

ERiC 
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(2) This step is a large cne. In the course of it you should 

List your own convictions in regard to questions such as 
those menticned ^ve; 

List your areas of incertainty; 

Review research reports on the pedagogical qi^stions 
involved: 

- to check on your certainties, 

- to relieve your uncertainties; 

Find out vhat media are on the market, and yihat they are 
designed to do. As to the '*new*' items, 

How new are they? 

Are they yet generally available? 

Who in your locality has used them? 

Do they really woik? 

(Have all the gremlins been found and 
eliminated?) 

But don^t overlook new media that may solve some of your 
problems. 

(In the tqpe-text language course Bonne Ecoute^ , the 
noimal speedi was too fast for students jusF" 
beginning to hear Frmch, so the tapes were put 
throus^ a speech conpressor operated in reverse. 
The syllables and pauses were lengthened, without 
affecting the pitch of the voice.) 

Get facts re comparative costs of various media, formats, 
materials, services, aid the time; involved in ^e 
production of them. 

6(A) Decide on Strategies 

These decisions you will have to live with throu^unit the develop- 
ment of the course, and peihqps for a long time afterwards. It 
would be very difficult to midce major changes in these broad strategies 
vAien far into the development process. Therefore, time should be 
taken here to consider thorou^ly the items noted in Step 5, their 
relationship to one .another, and their implications. 



* Bonne Ecoute is a tape-text program designed for adults \iho read Frendi 
fairly well and yiio desire to become equally con|>etent in listening and 
conversing. A first (e3q)erimental) edition will be ready for use In the 
spring of 1974. 
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6(B) List special services and equipmnt required ; 

At approximately vrfiat stage of the production process will each be 
needed? 

Will it be needed for the develq)mental version? 

(The closer the developmental version is to the planned final 
product, the more valid the develqmental evaluations may 
be*) 

Put the list sofiay for future reference. 

?• Write a rationale of the course 

This will pull your plans together. 

It will help to reveal any omissions or anomalies. 

It will help to keep you on the track during development, and serve 
as an introduction and reference source for people who will assist 
you. 

It will provide a start for the course manual, informing 

- those 1^0 siqpervise students, 

- those vdio decide vdiat course materials to purchase or adopt. 
8. Arrange the content in pedagogical order 

Which critical objectives depend on others? 
Which topics depend on others? 

Which •learning tasks" present more difficulty? 

(Here, a taxonony is useful - Bloom's Taxonony of Educational 
Cbjectives, Cognitive Domain* , or some variation of it.) 

E.g. These objectives are in order of difficulty: 

Select the correct (learned) name for a concept iiAien 
that name and others are given as choices; 

Match several concepts and their respective names, 
vihen concepts and names are given in separate randcm* 
order lists; 

Given the description of a concept, state its name; 



* Benjamin S. Bloom (ed.), Taxonony of Educational Obj ectives, Handbook 1 
CoflnltivB rwmi^n (New York: David McKay Conpany inc., 1956). 
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Given the name of a concept^ esqplain the ccncept; 

Given a set of circumstances ^ state the concept 
iMdi q^plies to them and ho^ it operates in regard 
to them. 

Should some prerequisites be reviewed in the course , before the 
content proper is introduced? 

Will there be footnotes ^ badoiotes^ a glossary ^ or will evetything 
^ be esqplained in the text? 

(Esqplanations in the text are more readable. 
Footnotes are easily available ^ and don^t take reading time 
if the student doesn't need them. 
Backnotes and glossaries are convenient as a repeated 
reference for any related parts of the text.) 

Must the content be learned in that order? 

9. Divide the content into lessons or units 

Where are the natural divisions of the subject matter? 

Is each unit to be a separate booklet , tape^ or other instructional 
entity? 

Would smaller uiits at the beginning be a motivational factor? 

What will be the approximate limit as to length of any one unit^ 
in teims of foimat? 

10. Write tentative objectives for Ihit 1 

(1) Begin with the critical objectives written in Step 4. Uider 
eadh one^ convert the ^'topics*' to behavioural objectives (sii)- 
objectives) . 

(Esqpress exactly vAiat you want the student to be able to 
do whm he conpletes the sequence leading vp to each 
sU)- objective* if necessaiy^ be specific as to how well 
he should do it; sometimes you may want to state a time 
limit as an indicator of his achievement.) 

(2) Behavioural objectives are conceived aid expressed in teims 

of iriiat the learner will be i^le to do - not what he will *lcnow** 
or 'Hiiderstand/', and not irfiat the program or instructor is 
attenpting to do for him. 

(3) A topic wa/ be difficult to express in behavioural terms » but 
if you can*t express it in those terms^ how are you going to 
test for mastery of it? The learner must show in some way that 
he has learned. When in difficulty you could say^ 'To demon- 
strate a knowledge of by .^^ 
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(4) The strategies you have selected should be kept in ndnd while 
writing objectives, in order to take fiill advantage of the 
media aid to keep the Uiit conpatible witii tiw instructional 
approach. 

(5) Writing objectives is tiaB-consuidng. At this stage the main 
thing is to ey^ress an objective so that knour exactly idiat 
you mm. If the perfect word escapes you, write anytidng that 
gives the meaning - you mof change it all around idien you write 
tiie material. 

(6) Keep the list of objectives for Unit 1 before you as a guide 
for the Uiit material. As you write an instructional seqtnnoe 
you mdoe much closer mental ccntact with the subject matter. 
Dynamics cooe to mind, interrelationships of topics occur to 
you, ways of relating the content to tiie student's eiqperlenoe 
and interests, opportunities for hummr that will li^iten the 
business of learning, ways of accx«|dishing two objectives in 
one sequence (or rather, in two diffeieht uses of it by the 
student) , the n^oessity for inserting another objective to make 
the chain of learning amplste - all these and more ideas will 
well up in your mind as you write. Many of them will result 
in alteration of the objective, or the way it is expressed. 
You need an objective to stan. with, but it is likely to 
develop with the Unit. 

(7) When the unit has been prepared, consider the objectives 
again: 

Were they intended for your use onlv? 

(If so, tiiey can be filed at tfds point.) 

Ifere they for quality control - to be referred to by someone 
who will edit the material to check for relevance, coqplete- 
ness, validity, etc.? 

(If so, the objectives should travel with the Itait 

draft.) 

Were they for an ir^tructor who will assign the Unit or 
monitor the studoit's use of it? 

(If so, they nay be expressed in teacher language.) 

Were they for the student? 

(If so, they should be expressed in student language.) 

Does a student need to see objectives? 

(If he has soae idea of idiere he's going, he has a 
better chance of getting there. If foraial objectives 
are not provided for him, substitutes for them should 
be incoxporated in the text at the beginniilg of the 
corresponding instructional sequences. There are some 
exceptions, but not in acadoidc areas.) 
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11. Write the instructional material for Uhit 1 

(1) Reneiii)er the purpose is not just to inform, but to help the 
student learn. 

(2) Dcn*t just tell him ithat to do, but explain why it*s worth 
doing ■■ how it will help him. 

(3) Picture a typical meni)er of the target population, and prepai« 
the material for him or her - or both - as if you were taUdng 
to tlie student directly. Be informal. If the student you 

\ picture has a sease of humour, tidcle it when you see a good 

> chance. If ^ don't delist in hunour, forget it: you can 
do only v^at you can do. 

(4) Ihe approach implied here does not involve the set patterns 
aid constraints of programed instruction, sudi as very small 
stros of similar length, one-word responses, or becoming 
lodoed into a linear or braidiing type of progression; and the 
word 'tedia" is not intended to include "teaching madiines" 
ndiich are merely medianical page turners designed to keep the 
student from peeking at the answer before he makes his response. 
Hie individualized materials will be '^rograBed", but in a 
mudi more flexible and informal way, with steps and sequences 
of varying length to suit their purpose. When you help the 
adult to learn how to learn, and let him use a learning sequence 
until to decides he has mastered it to tiie degree e3q;>ressed in 
the objective, the idea of '^king at the answer" becomes 
irrelevant. 

(5) Relate the content to the student's needs or interests. Try 
to give him some skills and experiences tiiat he can apply in 
his daily life. 

(6) You need to have in mind some taxonoiy of the levels of difficulty 
inherent in certain kinds of response demaided of the student. 
Bloom's taxonony of the cognitive domain* or some adiq)tation of 

it may be useful. The most-used levels are just camm sense, 
and should be observed automatically. Other levels mi^ be 
referred to vdien needed. However, using the taxonoii^ as a 
stea^ reference can lead to using too maiy levels for eveiy 
sequence; it is better to think of what would help thie student, 
and then dieck to see that the steps are in order of difficulty. 

(7) While the affective element is iii|)ortant in BTSD, a taxonoiiQr 
is not veiy xe levant. 



Bloom, op.cit . 
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(8) Above all, keep the language siople and direct. If reading is 
required, wake sure the level of difficulty is lower than 
prcbable reading ability of the students. In explaining a 
concept in Grade 7 nath, the explanation should not require more 
than a Grade 5 or 6 reading ability - and Grade 5 would be the 
better. If you don't know the readability of what you have 
mrittan, you can neasure it with the Fry readability naph* or 
sam sinilar tool. (See attached Annex.) 



(9) "Reviews" are useful but dull, aid are often skipped by the 
student. Ihe sane purpose can sometiaes be served by naking 
references to iteas previously covered, by providing 
incidental qiplications for them later, by emulative testing, 
etc. 

(10) Provide a self -test for the student at each main dieck point 
(eadi critical objective) and after most sii>-objectives as 
well. These self-tests m^ be eabedded in the text, or 
separate, or there may be some of both kinds. 

(11) ^If- tests, exercises, assignments, refietenoes, footnotes 
(badoiotes or glossary entries) should be prepaied before you 
leave the sequence to which they refer. It is easier while 
you have all the details in mind, and before you beame 
involved with another topic or another sequence of instruction. 

(12) As you prepare the material, ideas for the uiit end test (if 
there is to be one) will cone to you. Be sure to jot them down, 
along with the page nuBi>er wheie the related sequence is to be 
found. Don't consider the imit finished uitil the end test 

has been preparsd - it's difficult to come back to it later, 
aid you won't do as good a job. 

(13) Prepare descriptions of any audio-visual materials needed. 

List all items related to the uiit - books, tapes, slides, study 
cards, exercises, end test, etc. 

(14) As to your own qiproach to writing the materials. 

Keep the rationale in mind. 

Give your imagination and creativeness free rein. 



• This graph was developed by Edward Fry, of Ritgers University Reading 
^^lAiJ^f""^ P^lis^? in the April, 1968 issue of the Journal 
Of Heading (a pitlicaticn of the International Reading Associat ion) , wit h 
pBTBwsiai to copy. It appeared, with explaiations, in 'Here's How You 
Measure Readability" by Naida Waite (Caiadiai Vocational Journal, 
w ikTr * ^' several systems for measuring readability, 

but this Cne is e<nwHfl11v nrmtmniam*' '* 



but this one is especially convenient 

1J6 
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Be receptive to insights and inspirations vAiidi will 
one to you as you gp deeper into the stfcject - even if 
these mean changing iriiat you have just written. It's 
easiest to change now. 

If you have enoudi training and experience to develop a 
course, you ^ould heed your own intuitions and coomcn 
sense vhere they conflict with some current system. 

The hardest word to write is the first one. Write something 
aid get started. You'll go back and change the first part 
anyway. 

If the well of inspiration goes dry, just picture that 
imaginary student you are writing for - iduit use this bit 
of content will be to him, aid how you would help him to 
leam and appreciate it if he were sitting beside you. 

12. Ite -examine the Ihit 1 draft after an interval 

Put the draft away for a few days , uMle you start Uiit 2 or do 
something else. Then re-examine Unit 1. 

Are the objectives clear, concise, precise? 

Will the instructional sequences help the student to achieve 
the objectives? 

Are there any omissions? 

If something was changed, was it dianged in every place ndiere 
it was involved? 

If the student was told where to find the answers to an inbedded 
self-test, are the answers there? Are they correct? 

Have the selected fomats been adhered to? 

If art woik, audiotapes, slides, etc., are needed, are there 
adequate instructions as to format and content of these? 

13(A). Draft the next few uiits in the same way 

13(B). Ensure availability of resources 

Ensure that any special staff, services or equipment will be 
available at a stated time nAien they will be needed. 

14(A). Have a few units produced for developmental evaluation 

You may want to produce these materials in some quick, economical 
way. "Printed" materials m^ be duplicated by some cqpying 
process. If audiotapes need to be dif>licated, you may hook 19 
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a few tape recorders instead of sending the master tqpes 
awayr for this service. If art work is needed, you use 
pictures out of magazines or from other sources for this first 
trial rui. 

1*(B) . Arrange for a few representative stud«its to try the materials 

Develcpmsntal evaluation involves only a few students, working 
with the materials in a location \diere you can conveniently 
monitor their progress and reactions. 

These few students should be carefully selected, and should be 
as representative of the target population as possible. 
(Ca 'er age, sex, ethnic backgrouids, and learning 
*i y as evidenced by some standard test; watdi cut 
for poor hearing and poor eyesi^t, and at least be 
aware if a student has a deficiency; one or two such 
students out of ten or twelve mi^t make the sample 
more representative, provided they can cope with the 
materials.) 

If the course is long, and each unit d^nds on the mastery of 
previous tnits, developmental evaluation requires students who 
will take the vtfiole course. If you start the evaluation with 
too few units conpleted, the presence of these students may put 
pressure on you to produce more uiits as they are needed; you 
can't afford to let such pressures jeopardize quality. On 
the other hand, if you leave it too long, you will have created 
too mudi material without coming to grips with reality as to 
its value; it will be e3q)ensive to change, and you will be less 
likely to make needed changes. 

If you can use students vtfio are not mder some financial 
coniiulsion to be in attendance every day, and vho are not con- 
strained by a time limit, you can excuse them from attendance 
if they catdi up to you, or you can have them do something else 
for a day or so. They are there for your convenience, but their 
presence can create much undesirable pressure if you are not 
in a position to relieve it. 

You mi^t begin with very few students, and take in more vtfien 
you are read/ to re-evaluate units that have been revised 
following the first trial. In this way you will probably always 
have a diance to re -evaluate any unit that proves troublescme. 

How will you recruit the students? 

What will their schedule of study be? 

Full or half d^s? 

At what times? 
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What will their study environment be? 

Will the activities of some distuib others? 
(Earphones may be necessary for s<»ne 
voice-reproducing media) 

Will Uiey be permitted to consult each other? 
Cniey should y if this is to be permitted 
in the final course dynamics.) 

What rewards are involved? 

Do the students just get a free course? 

Will they receive some financial reward? 
If so, fran lAom, arid implying 
\ihat constraints? 

Will videotapes 9 audiotapes, or other visual and/or 
auditory records of this group be used in subsequent 
instructor-training courses or in explaining the 
course to the public? If so , a form of release or 
agreement to this should be signed by the prospective 
student before he is accepted in the developmental 
groi4>. 



You may have the students hand in all their written work, if 
you have time to look at it. 

At least, you will want to observe the completed self -tests 
and unit &)d tests, if any. 

Spend some time sitting quietly in the room to observe how 



Are there any signs of boredom, fatigue, frustration? 

Do they keep working steadily? 

Do they handle the A/V equipment efficiently? 

Do they succeed in synchxxmizing their use of 
two media if required? 

Do tiiey follow the instructions in the units, as 
to the way the materials are to be used? 

You might go in as the "instructor" or •'helper" (not as 
the developer, if you can avoid it), and ask the students 
to bring their difficulties to you for help. 



15(A). Conduct the developmental evaluation 
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You might talk to the students to get their reactions. 
However this usually brings only kind remarks. A 
question such as "What part of this unit did you have 
to work the hardest on?" might point to some sequence 
that needs revising. 

You might have an evaluative questionnaire completed 
by the student at the end of each unit. The student 
may be more frank if this is anonymous, but it will be 
more useful to you if it is identifiable. Real anony- 
mity is not often achievable in a small group. 

You migjit tell the student that the unit he has just 
completed may be changed, and ask if he has any 
suggestions as to how it can be inproved. 

How much did the student actually learn from the unit, 
that he didn't know before? 

Where there is an end test with many responses required, 
you may want to use the "Modified Gain Score"* as a measure 
of gain, as follows: 

(1) Give the end test as a pre-test. 

(2) Give it again as a post- test. 

(3) Possible Score minus Pre-test Score equals Possible 
Gain. 

(4) Post-test score minus pre-test score equals Actual 
Gain. 

(5) Actual Gain x 100 » Mviifipd na^n c>«^-» 
Possible Gai n Modified Gain Score (I) . 

15(B). Keep a list of worthwhile changes and suggestions from any source 

Minor changes to be made before materials are produced for 
field testing. 

Major changes that must be made in the developmental version 
for imnediate re-evaluation. 

Snail omissions from early units may show up as you are 
developing later units - perhaps a tenn you wish you had 
introduced if you are using the spiral curriculum approach. 
Deal with them in the later units if possible - chance the 
early units only if necessary, or very simple to do. 

* me Hodiiiea uain Score has been used at the State Ihiversity of 
Nebraska, and it may have originated there. 
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Keep a list of changes desired but not made, and add to 
it as further work and Inspiration bring possible improve- 
ments to mind. 

Some minor improvements of format you can incorporate in 
the current and future units, and list for incorporation 
in the completed units when the revised edition is prepared. 

16. Revise as necessary for immediate re- evaluation 

Only major changes call for re-evaluation: 

E.g., the complete rewriting of one or more 
sequences; the addition of a sequence, 
an exercise, etc. 

17. Re-evaluate units with major changes 

Use new students if possible, but you could also have 
those viho used the first version try the naf one and 
comment on it. 

18. Recycle as many times as necessary 

If the second version doesn't work, you might check to see 
if all prerequisites have been provided for the failing 
sequence. If the problem is failure on the unit end test, 
you mi£^t suspect the test itself, and also check to see 
if the unit has provided for enough review. Is the unit 
too long, in terms of time required to complete it? Is 
it boring? Is it relevant to the student's needs and 
interests? If there appears to be no general cause of 
failure, rewrite and re-evaluate again. 

A unit on '^ordering from a catalogue'* mij^t prove 
to be a "drag*' for the men in a literacy class, but 
quite acceptable - though difficult - for the i#onen. 
^is is a task a husband is inclined to leave to 
his wife. Perhaps that unit should be considered 
optional. If two units are very similav and indepen- 
dent in nature, they should be separated in the series 
to avoid boring the learner.) 

19(A). Develop the rest of the units and prepare the general items 

Ge;ieral itans may include: 
A course manual 
A book of answers 

A special tape for some purpose such as 
practice in listening, practice or tests 
in spelling. 

A book, tape, or coni>ination of media to show 
the student how to use the course or operate 
the media involved. 
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It is important that users be able to find a specific 
topic in the course materials irfien they want it. A 
detailed table of contents in each booklet will help - 
but i*y not put it on the cover ndiere it can be seen with- 
out having to open the book? A list of critical objec- 
tives, or all objectives, placed in the manual, is useful, 
but it mi£^t well be followed with a "content analysis" in 
sumnary form, ndiich could be skinned throu^ more quickly. 

It may never be ne ssary to include an index, and it 
certainly would not be practical until the course materials 
are in their final form, after the field test. 

19(B). Plan for the field test 

(1) Studmts should be representative of the target 
pq[Mlation. 

Consider age, sex, ethnic backgrounds, 
learning ability as guaged by some standard 
test. 

If the course is to be used in a wide geo- 
graphic area, several different parts of 
that area should participate in the field 
test. 

(2) How many field-test students do you need to give a 
significant evaluation? How many students, in what 
geographic locations, can be monitored efficiently? 

(3) How much work will the feedback process entail for 
the field instructors or st^rvisors, and for your 
own office? 

(4) If instructors will be involved, you will need to 
inform them of the purpose, principles and nature 
of the course, and ensure that they will be able 
to use the materials in the manner intended. Much 
of this can be covered in the course manual. 
However, it will be more helpful to have a familiariza- 
tion soninar, with opportunity to handle the materials, 
use the media, and see demonstrations of some of the 
processes. Such a seminar is almost necessary if the 
course is an innovative one. 

(5) Those vdio will monitor the field trial will need to 
understand the course, as well as vAoit sort of feedback 
you want, and how you want it reported. 

(6) You will want feedback as to how well the course is 
liked, in addition to its effectiveness and efficiency. 
If you can arrange to talk with some of the students, 
as well as some of the instructors, tutors or super- 
visors, this will be very helpful. 
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(7) You may provide a stati5cical form for feedback, but 
this should be siqpplemented from the field by infor- 
mal assessments and anecdotal reports of particular 
significance. Statistical reports should shau the 
progress of each individual student, along with signi- 
ficant personal data about that student. (Summary 
statistics alone are almost useless to the course 
developer; much valuable information is buried in totals, 
averages and percmtages.) Personal data may reveal 
significant groi:^ teiidencies or preferences. 

Personal data should' be reflected in any statistical 
reports - e.g., brealodotms according to age, sex, 
language spoken at home, etc. 

(8) You might have some of the students* work <m unit end 
tests sent in, but if you hope to handle much raw 
data you will need to limit the numbers involved or 
blew that you can get plenty of help in sorting, re- 
cording and analyzing the data. 

(9) Drop-outs are of particular interest in a field test, 
but it is difficult to analyze their implications. 
Many drop-outs don't give a genuine reason, and some 
just disappear without giving any reason, so an attenpt 
to record reasons is probably useless. However, if 
same or similar content has previously been presented 
in the same area by means of another course or approach, 
the two drop-out rates for similar periods mig^t be 

(Mike sure that field-test students achieving 
early completions don*t get counted as drop- 
outs!) 

(10) People sometimes assume that a field test should include 
a control groiq;> for ccnparison with the e3q;»erimental 
group, but often this is not very useful. If the new 
course is truly innovative, it may be capable of producing 
results and stimulating interest not possible with the old 
course, and therefore there is no meaningful basis for 
comparison - especia).ly, if the final criterion is a test. 
In this case you can hardly test the control group on 
items they haven't had a chance to learn, and if you use 
a control groim test for the experimental groi^, they 
won't have a chance to demonstrate the extent of their 
siqperior gains. It would be more useful to compare the 
average length of time taken for ccmqpletion of this and 
some previous courses with similar content ; also the 
percentages of completions and drop-outs, if this infor- 
mation can be obtained. Ihe most inqportant thing is to 
find out if the new course works well, and what specific 
amendmmts, if any, should be made. "Working well" includes 
appealing to students as well as being effective and effi- 
cient in helping them to learn. 
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20. Have the field test edition produced and distributed 

Don't hold field test material back to make small changes. 
Put these changes on your list for later attention, and 
let the material go to those i^o will be producing it. 
The main thought now is to start the field test as soon as 
possible. 

At this point you want the materials to be as close as 
possible to your idea of their ultimate form. During the 
final development stages you will have been receiving actual 
items of art work or other media, and substituting them 
for any makeshifts used in the developmental evaluation 
stage. At this stage, the materials will probably be pro- 
duced by ccHimercial or in-house printers, tape duplicators, 
etc. Printers will probably require a "dumy" for each 
booklet involved. This will be a duplicated version, 
including a damy cover and the pages showing publisher and 
copyright information, inside title page, etc. If the text 
is to be printed on both sides of the paper, the pages will 
be taped back- to -back, the odd numbers always being ri^t- 
hand pages. The printer may also require sunmaries as to 
the number of booklets, total nunnber of pag^s, etc. , and 
he will need infoxmtion as to stapling or binding and 
other details of production. 

In the case of audiotape cassettes, commercial duplicators 
dt5)licate both, (or all) tracks at one time. If the masters 
were to start at the beginning of each side, the total 
length of the tape would be used; if a blank used for di;pli- 
cating is slightly shorter (and they do vary) , some content 
will be lost. The safeguard is to stop recording the first 
side well short of the end; turn the cassette and start the 
second side at that same place on the tape. 

21. Conduct the field test 

Ideally, someone should monitor the field test operation - 
you, or someone vtfio is fully familiar with the comrse and 
the way it should be used. 

Is it being used as directed? 

Are the students representative? 

Are there any serious environmental elements 
militating against it? 

Is any serious defect showing up generally or 
in regard to any specific group? 

What are the reactions of instructors, tutors, 
supervisors, or whoever is the immediate contact 
for the student? 
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22, Analyze feedback from the field test 

Use very lead you can get to assess the value of the 
course and decide vdiich parts or vihat aspects, if any, 
need changing. 

Study the statistics, student work, suggestions from 
the students and from the instructors, tutors or super- 
visors, the observaticms of vAoever monitored the field 
test, and consult your notes of any conversations you 
have had with those involved, and any other feedback 
sources. Some irn)ressions will conflict, and will have 
to be weighed carefully. Some suggestions, vAiile good 
in a sense, may be conpletely out of character for the 
course. 

Use the feedback, but filter it through your own evalua- 
tive processes , keeping in mind the aims and rationale 
of the course. 

23(a). Revise the fie Id- test version 

It is unlikely there will be man>' major changes at this 
stage, but if a major change is necessary, this will mean 
recycling through several previous stages. Developmental 
evaluation will be a problem, unless you are rewriting the 
vdiole course. You may have to look for a few students vfco 
have forgotten the specific - subject matter you are re- 
vising, and give than a quick review of the prerequisites. 

At this stage you will consider the small changes you have 
been listing as they occurred to you, and incorporate or 
adapt those that still seem desirable. 

As you read over the units at this stage, you may want to 
make the tone less formal in some places, make a sentence 
clearer, use a shorter word or e:q>lain the long one,- 
change an exain)le that proved perplexing to some, etc. 
Uiless the vrfiole course is being retyped, retaped or other- 
wise redone for some general change of format, it may be 
possible to make these small revisions in the field-test 
originals . 

23 Cb). During the revision process you will need to consider the 
packaging of the revised edition. 

What kii^s, shapes and sizes of items must be included? 

Do these need to be in related pairs or sets within the 
kit or package, or will they be organized according to 
the kind of medium, or according to size and shape? 

Will the container be used for storage and retrieval 
viien the course is in use? 
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Who will retrieve and put away - the student, or an 
instructor, tutor or siq[>ervisor? 

Will labels need to be designed and printed for tape 
cassettes or other non-print iteois? 

Who will put the labels on? (The producer is probably 
the best - e.g., the person duplicating the audiotape.) 

How w?ll the items be minbered? (In a large course, 
each separate item should have a short coded designation 
of some kind, for reference, printing control, stock 
control, ordering, etc.) 

How mi^t the container's be designed and of what »aterials 
mi£^t they be made? (Suppliers will be helpftil in giving 
estimates of cost and suggesting inprovenents in the design.) 

23(c). Consider volume to be produced 

Estimate the probable demand over a given period * 
e.g., two years. 

Consider available storage space for stock on hand. 

Consider the cost of stock piling in relation to the 
budget. 

* 

24. Have the revised edition produced and disseminated 

IXiring the first few years, you may want to know \ihere 
the course is in use, and you can probably make some 
adminis iTative arrangement to receive this infonnation. 
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GBIERAL QUESTIGNS AND RB1ARKS 



How long does it take to produce a course? 

It is difficult to foresee. Ihis kind of develofnent is in the 
nature of action research » and, like any research project , the 
date of its successful conclusion is not predictable. Cm you 
estimate the developmmt time? Sometimes you have to do so , but 
you'll probably be wrong. The first five units may average 10 
days each, and the sixth one nay take 30. Besides that, all sorts 
of unforeseen things will happen - people you depend on will have 
to leave or be absent , machines will break down, occasionally 
something will get lost, sometimes the well of ideas and creative- 
ness will go dry, sometimes you have to stop to sleep and eat. 
Vfy personal advice to the developer viho is asked how long it will 
take, is to make as reasonable an estimate as possible and then 
multiply it by 5 or 10. It doesn't matter ^ich - either one will 
be wrong. An alternative is to decide you will take just so many 
days to create each init, and then keep to the sHieSule; in this 
viay you grind out the units like sausages y and they show about as 
much evidence of creative imagination. If your supervisor needs an 
estimate for administrative purposes , it is ijiportant to let him 
or her know that a close estimate is not possible. Dcm*t lead the 
administrator into premising or advertising a con|>letion date that 
may not be attainable. 

How do you work with an assistant developer? 

Sometimes the situation dsmnds that a course be conpleted by a 
certain date. You must do it in that time or not at all. Few 
people » especially early in their careers would opt for not doing 
it. If you decide it must be done in a short time, then you will 
need one or more persons to help you with the actual creaticn of 
the materials. (Jhis is totally different from having others help 
with certain parts of the effort iriiile you remain the sole creator.) 
Let* 5 snr you divide the course into two levels » and that you will 
create the first level and your colleague the second level. How 
can you make the best use of the colleague's abilities and keep some 
coherence between the two levels of the course? Here are some 
suggestions: 

(1) Recruit someone v4io has pedagogical knowledge » experience and 
creative ability^ and \iio also has a. good command of the siib- 
ject matter. If you have to do without one of these » it had 
better be the experience. 

(2) Discuss the project with the recruit » both to acquaint him or 
her with it» and to explore how well the recruit's ideas and 
convictions fit with the ideas and convictions by i4iich you 
hope to shape 'ie course. If the fit is reasonably good» 
select the recruit; if not, look further. 
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(3) If all of the basic decisions about the course have not al- 
xea^ been made, include the colleague in the decision-making 
process. B)r all means, make the laying out of the topics a 
joint effort, fbr both levels, so that each will him what 
the other will be doing. Share this effort, but reneiriber you 
are in charge of the project. You are responsible for the 
outcome, and you make the final decisions. 

(4) Draft the first uiit or two of your level , and discuss the 
drafts with your colleague, to establish approach, "tone", 
foimat, etc. 

(5) Have the colleague show you the draft of his or her first uiit, 
and discuss it. Have the first one or two mits worked over 
tntil they satisfy you. This doesn't mean that they will be 
identical to vdiat you would have done, but, rather, as the 
other person's out^t, they achieve a standard yibidi you would 
be willing to accept for the field test edition. 

(6) When you are satisfied that the colleague faiows what you want 
and can create material acceptable to you, let the colleague 
go ahead and you do the same, at your respective levels. The 
colleague may consult you, and you will Icok over the conpleted 
units. You may show your completed units to the colleague if 
he or she desires this, and if there is time. Otherwise the 
two levels may have to be created in a large degree of separ- 
ation. In any event, the colleague will be creating the laiit 
vdiich follows your last one, long before your last uiit is be- 
gin. 

(7) How will the colleague obtain developmsntal feedMck, when no 
students have yet conpleted the first level of the course? 
Some conpronise will have to be made hep^^ : rnaybe one or two 
students can be found ylho have covered the content long ago. 

A quick tutorial review of the first-level content may prepare 
them to woik with the second level tiiits. 

(8) While the course is out to field test, go over the Level 2 ma- 
terial, and make notes of changes that will make it cohere 
with your own material - add a reference to scne Level 1 item 
here, a touch of himnur there, bring the format better into 
line, sii)stitx2te some preferred approadi or dynamic, and so on. 
In other words, note tiie changes that will make it yours. 

(9) When the field test is conpleted, proceed to amend both levels 
yourself. If credits are being given, you can give your col- 
league credit for vdiat he or she has dene. 

Hw does anyone know how to write a course? 

It is helpful to have been present vAien a course was being designed - 
to know at first hand something about laying out the topics, se- 
quencing them and writing behavioural objectives, and to be prepared 
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psychologically for some misadventures. 

The preliminaries are in|)ortant» but the course comes to life in 
the actual creating of the materials idiich the student will use. 
Here the course writer must be guided by principles that have be- 
come almost automatic; such as these: 

Go from the familiar to the infamiliar. 

Go from the easy to the more difficult (and make 
sure which is idiich) . 

Prepare the mental set of the student for v^t 
he is about to mdertake. 

Don^t present easily confused items at the same 
time - separate them well in time and space. 

Don^t use '^elegant variation" in language - al- 
ways use the same word to represent a thing or 
idea intil it is well established; then, if there 
is a generally accepted alternative wording, it 
may be introduced and used interchangeably with 
the first; otherwise, Bt«f with the original 
wording. Ihis 1^7 mean using a word three times 
in one sentence} do it. 

Keep the language within the assumed or estab- 
lished reading or listening ability of the re- 
presentative student. 

If » in using ^ihat he is learning^ the student 
will neeSTraidom access to it, make sure that he 
achieves random access to it during the learning 
process, and that he knows why he*s doing it. 
(For instance, he shouldn't have to go throi^ 
the whole "7 times" table to arrive at "73c9".) 

Test the student on only what he had an opportun- 
ity to leam. If you want to give him a diance 
to "discover" or "apply" or "extnq)olate", do 
this in the learning sequences . Then you may (or 
may not) put an item of this kind in the test 
lAiich follows. 

Keep vp your reading of jouxnals and research re- 
ports in your field. It's hard to do two or 
three years' reading i^en you start to develop a 
course. 
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Be a student yourself from time to time, learn ;:ig 
sowBthing that really taxes your capabilities. 
Otherwise, how can you maintain any personal knowl- 
edge or feeling about the learning process and 
the business of being a student? 

Never engage in one-ip-nanship - don't play cat- 
aad-DDuse with the student. Keep your role as 
helper f iimly established in your mind and his . 
If there is a joke, he must know you are lausM^g 
too, and at the same thing. 

Never tell him something is going to be easy; if 
he misses the first time, he'll think he's stn)id; 
if he succeeds, you've spoiled his satisfaction « 

Provide distributed learning - by a spiral curricu- 
Im zppTOBidhy dealing with a topic several d'if- 
ferent tiines, on increasingly sophisticated levels* 
Also encourage the student to distribute his learn- 
ing efforts over several moderately long sessions 
rather than one very long one, and to t^e 10- 
minute breaks occasionally. 

Define any technical words you must use , show how 
to pronomce them if this is not obvious, and give 
exan|)les of their use. 

Sometimes you must use a difficult non-technical 
word because it is one generally used in that 
context* A formal definition mi{^t insult the 
student >dio knows the word, and you can't assime 
another student will look it up if he doesn't 
know it. You can usually introduce the word in 
such a way that its meaning is made clear without 
a formal definition. 

Keep the language clear, simple and direct. 

Keep that imaginary student beside you, and com- 
municate with him. 
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ANNEX 

THE FRY READABILITY GRAPH 



This graph, ^veloped by Professor Edward Fry, of the Rutgers University 
Reading Centre, is r'^tul in estimating the difficulty of reading materials. 

It is based on the assumption that long sentences and long mords make 
reading difficult. The results are approximate, because some short technical 
Js may be unfamiliar, and sane short words may be combined into unfamiliar 
iii^<ises. However, this has proved to be a very useful and convenient meax.:: 
of estimation. 



The process is as follows: 

' . delect three passagej near the beginning, middle and end of the 
material, and count a 100-word sanq)le in each passage. Iimore all 
proper nouns. 

2. Count the mnber of sentences in each 100-word sanqple. If the sample 
ends with an incomplete sentence, estimate the included part of that 
sentence to the nearest tenth of the sentence. Find the average of 
these numbers for the three samples. 

3. Count the total minber of syllables in each lOO-word sanq;>le. (You 
count one syllable for each vowel sound that is pronounced. For 
instance, '"bat" has one syllable; watch, 1; through, 1; over, 2; 
inches, 2; thanselves, 2; magnetic, 3; easily, 3; continental, 4; 
dictionary, 4; foimidable, 4.) Find the average of these nunbers 
for the three sanqples. 

4. On the graph, find the horizontal line that represents your average 
nunber of sentences per 100 words; then fin^i the vertical line that 
represents your average lainber of syllables per 100 words. Place a 
dot ^ere the two lines cross each other. Read the grade level for 
the section of the graph yAiere the dot falls. 



(Please see the graph on the following page.) 
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GRAPH FOR ESTIMATING READABILITY 

by Edward Fry, Rutgers University Reading Center 
Average number of syllables per 100 words 
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&q)lanation : (a) The more sentences you find in 100 words, the shorter 
the sentences ,^re, and the easier the reading level is. Therefore, Grade 
1 is near the top of the graph, and "college" near the bottom. 

(b) The fewer syllables you find in 100 words, the smaller 
the words, and the easier the reading level. Therefore Grade 1 is near 
the Irft of the chart, and college near the right. 

(c) If the dot falls far to the right of the lines that 
crosj the main gr^h line and above them, you have very long words in 
veiy ohort sentences; if it falls far to the left and below, you have 
veiy long sentences made up of very short words. In either case, you 
might consider the material to be unsuitable for classroon use. 
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Su ggestion ; Counting the hundred words is easy, buc counting the 
syllables needs a different strategy. One almost has to pronounce 
the words mentally to be aware of the syllables , and one cannot be 
counting at the same time. A convenient method has been to put a 
finger down for each syllable and tally^^e fives in the usual way 
{Hhf+h ). Then the syllables can be totalled from the tally. Tliis 
tally C //// ) would represent 25 syllables. 

Credit ; The graph was published in the April 1968 issue of the 
Journal of Reading (a publication of the International Reading 
Association) . The ^uthor and publisher generously gave permission 
for free use of t^ ;raph with suitable credit. 



QiAPTER VII 



GOURSE DEVELOFMEKT 
FRCM IHE 
AEHINISIRAIOR'S VJEMPOINT 



Naida Waite* 



Designing and devel(q)iiig a ccnq>lete course is not the kind of 
project that can be done by a teacher in spare evenings . In addition 
to professional expertise, it requires clerical ai^ technical assist- 
ance, money and time. We'll come back to the siiject of time. 

In the following remarks regarding course developnent, we are 
talking about an individualized course. Here, the course materials 
have the main responsibility for presenting the content and helping 
the student to learn. Ihe new individualized materials are designed 
in a flexible, infoimal way, and provide a variety of learning activities 
or ^mamics. They help the student to 'learn how to learn, and let him 
use a learning sequence until he decides he has mastered it to the 
degree expressed in the objective. He works at his own pace, skips over 
what he already knows, and asks for special help when he needs it. The 
teadier or tutor becomes a guide and resource person, diagnosing dif- 
ficulties and helping the student overcome them, and acting as the final 
judge of achievement. 

We are also talking about a subject area that is highly structured, 
such as literacy or a second language, where sequential learning is 
ij^wrtant . 

Ut's take an overview of the total development process for that 
kind of course. Here are the main steps. 



Specialist, Basic Literacy Programs 
Training Research and Development Station 
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COURSE EEVELOP>CNT (HEa LIST 



List tenns 
of 

reference 




List leamer 
diaracteristics 
and needs 




Investigate 
instruction 
already 
available 


If it is 

suitable. STOP HERE 




possible, consider other 
priorities 


; musable, GO PHEPS) 



List content 

(1) Tentative 

(2) Necessary 
in form of 
critical dbjec 
tives with 
"topics" uider 
each 




Consider strategies: 


Search out 


Instructional 


relevant 


sqpproach 




Student's d^andcs 




ffedia 




Evaluation procedure 




(Budget) 




etc. 






List special 
services and 
equipment 
required 



Write a rationale 
of the course 



8 

Arrange the 
content in 
pedagogical 
order 



Divide content 
into lessons 
or inits 




10 

Write tentative 
d>jectives for 
Uhit 1, convert- 
ing "topics" to 
sub-objectives 



11 

Write instruc- 
tional material 
for Uhit 1 




12 

Re-examine the 
lliit 1 draft 
after an 
interval 




]m 

Draft the next few units 
in the same way 



Insure availability of 
special staff, services or 
equipment to be required 
at a stated time 
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im 

Have a few uiits produced for 
developmental evaluation 



14(B) 




Arrange for a few represen- 
tative students to try the 
course i4iere you can monitor 
the process and results 



15(A) 

Conduct the 

developmental 

evaluation 




Keep a list of 
proposed changes 
minor - major 



17 

Re-evaluate 
units with 
major 
changes 



18 



19(A) 



20 



Recycle as >v 
many times as \ 
necessaiy 


Conplete the development of ^ 
the course - remaining tnits , 
graieral items, etc. ^ 






19(B) 






Plan for the field test 






19(C) 





Gradually substitute actual 
art work, etc. for any 
makeshifts used during 
development, as these proper 
items are received 



Have field test v. 
edition produced \ 
and distributed 



21 

Conduct 

field 

test 



22 



Analyze feedback 
from field test 




23(A) 



24 



Revise as necessary or desired, 
and evaluate any major dianges 




Have revised 
edition produced 
and disseminated 



Consider packaging of Revised 
Edition 



23(C) 



Consider volume to be produced 
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If suitable individualized'^itterial can be found, it is not prac- 
tical to launch into the development of a new course. But how do we 
interpret the word "suitable*'? No two cooks make their beef bourguignon 
in exactly the same w^, but the clientele may be equally pleased with 
botE^ Rotwo teachers teach in exactly the same way, but both m^ be 
effective. A teacher will never find an individualized course that 
covers exactly the desired content in exactly the way he would teach it. 
He may have to ccrabine the relevant parts of two or more courses, and 
find out if the individualized material will do the job expected of it. 
But we assune that the student will have access to a teacher, tutor, 
or some helper, \dio will provide further e^lanations vihen needed and 
suggest further learning tasks. Therefore, the question is, *Vill the 
course do the expected job with a reasonable amount of sipervision and 
help?" 

Let's assune we have decided to develop a complete course of the 
individualized kind referred to, vdiich, if it had been presented in a 
traditional classroan, would have been scheduled to occiqiy 9 or 10 months. 
Let's say it will contain printed words and other media. The questions 
that immediately come to mind are: Who should develop it? What about 
media? How should the course be evaluated? What if a unit or sequence 
fails to work? How long will the development take? What will the total 
cost be? 

We'll start with 'Wio". The developer should be someone vAio knows 
the accepted principles of learning (and let's be concerned with learning, 
rather than teaching) , keeps abreast of current research in education 
(and is able to distinguish iisefiil research fron the trivial and ill- 
conceived), is not afraid to innovate, has a lively imagination and a 
way with words, appreciates what the various types of media can do, 
and is perceptive as to students* difficulties and interests. To have 
been present \ihen a course was being designed and developed would be 
an advantage, especially in laying out the topics, sequencing them, 
writing behavioural objectives, and being prepared psydiologically for 
a few misadventures. Continuing to be a studlent from time to time in 
areas which tax the cq[>abilities would also be helpful, for how else 
can one maintain a first-hand knowledge or feeling about the learning 
process and the business of being a student? 

There are many media to select from, and more are constantly 
gearing on the market. How many different media should be introduced 
in a single course? If we want to use ten different kinds of media 
that will best produce them - one for each. But two or three media, 
including print, will be our J.imit, for we want to avoid inconvenience 
to users, the conqplexity of many different machines, and unnecessary 
expense. With ingenuity, we can iisually make one medium perform almost 
the same function as another. For instance, mai^ different ^oiamics 
can be acccnq)lished with the tq;)e-text ccnbination. 

While the newest inventions my be plagued with delays in delivery 
or with perfoimance problems that haven't been overcome, it is neverthe- 
less important to keep infoimed of new products. A new kind of mediun, 
or a new feature in an old one, may solve an otheiwise difficult problem. 
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For instance, the voice conQ>ressor, whidi changes the speed of speech 
withcut altering the pitdi, solved a problem in a second-language 
program. The ordinaxy speech as recorded was too fast for novices to 
distinguish the pronunciation clearly, so the con?)ressor was used in 
reverse to slow down the speed. 

How should the course be evaluated? Staall-scale evaluation should 
be conducted £ram day to day during the development process, and a 
lai:ger-scale field test should be used to evaluate the completed course 
as to its effectiveness, efficiency, student appeal, administrative 
convenience and so cn in several locations and environments which 
represent its intended target population. 

The developmental evaluation is done with a few students - s^ 
ten or a dozen - viio use the materials in a location viie ; the developer 
can conveniently observe student reaction, the learning process and 
results. This enables the developer to keep in close touch with reality, 
shaping future units on the basis of experience. It helps to avoid 
costly major rewrites, and it shows up difficulties at an early explor- 
atory stage while the developer is still quite willing to make a change. 

Since individualized instruction facilitates continuous intake, 
the few students used in development might well be enrolled in two or 
three stages, in the hope that some would be coming along behind the 
others and be available for the second trial of a unit that has to be 
rewritten. 

The field test will not require the inmediate presence of the 
developer, although it should ideally be monitored by the developer 
or someone **io knws the rationale of the course and is familiar with 
the materials. If the coui^e is innovative, preliminary training 
should be given to the teadiers, tutors, or vdioever is to help the 
students on their way throu^^ it. 

Should there be a control group with which the field test grou^ 
could be conqnred? In many cases this attempt to be scientific merely 
clouds the issue. Let's remenber we have developed this course because 
nothing already on the maiket was suitable. We went througji the dev- 
elopmental evaluation in order to sh^ our course into something that 
would work. Han we want to expose the materials to more and different 
students, different teachers or tutors, different geographical locations, 
and the absence of any super-noimal effort on the part of the students, 
vdiidi might have occurred in the case of students who helped with the 
developmental evaluation, through regarding themselves as a chosen, 
elite groiq). The immediate purpose is to see if this broader exposure 
will reveal any shortcomings that didn't show up previously, and to 
receive suggestions from those who represent the ultimate users. The 
aim is to gather information on vrtiidi to base revisions for a final 
edition. 
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And let's remember that a great deal of important information will 
be of the anecdotal type, which is lost in the totals, averages and 
percentages of tabulated hard data. We need both kinds of feedback, 
for we must know hew well the students like the course, in addition to 
how well it helps than to learn. Drop-out statistics are meaningful, 
but reports as to students' enthusiasm and initiative in making use of 
vrtiat they have learned are also helpful. We need to know if a specific 
sequence has been found difficult by many students, if some reference 
has proved incomprehensible to students in some area, £ some features 
of the materials have proved inconvenient. Every possible means should 
be taken to gather useful information from the field test. 

Let's go back for a minute to the developmental evaluation. What 
happens if a unit or sequence doesn't work well enough? It has to be 
redeveloped until it does do the job expected of it. It my have to 
be recycled througli the development and evaluation stages more than once. 
But let's remenfcer this is action research. The developer, with an open 
mind, is trying something ouc - testing a hypothesis, -if you like. 
There are good reasons for expecting that it will work, but no guarantees. 
The developer who is exploring the impact of a new approach has a right, 
at this stage, to be wrong. Uhless ^is is clearly imderstood by all 
concerned, including the developer, there is no point in expecting 
original or innovative output. A unit rarely fails completely, but 
some decisions for change may necessitate a complete rewrite, and a 
rewrite necessitates recycling. 

That brings up the question as to how long the development will 
take. This is difficult to foresee. It is action research, and, as 
with any research project, the date of its successful conclusion is 
not predictable. If the situation would dictate a very early deadline, 
it might be better to make do with some barely usable materials, and 
either postpone the development of a new course or let the development 
process occupy whatever time is required to conduct it properly. 

If, for administrative purposes, it is necessary to estimate 
development time, we might make a reasonable guess arid then multiply 
it by 5 - or 10 - to allow for staff absences, staff training, problems 
in finding staff, machine breakdowns, delivery delays, recycling of 
revised material, occasional misplacement of papers, library searches 
that must be made to ensure the exactitude of a reference, times when 
the well of creativeness runs dry through over-use, the endless checking 
and rechecking of prepared materials, and the time spent describing the 
project to visitors vAo have made a journey to hear about it. 

Sometimes the developnent time can be reduced by providing one or 
more junior professional assistants to the developer, in addition to 
the clerical and tedinical people alrea(ty mentioned. These assistants 
may do detailed work such as looking up references, drafting exercises, 
helping with the developmental evaluation, tape recording, and so on. 
This arrangement is convenient and extremely helpful. The developer 
and the professional assistants work closely together on the same lesson 
or unit as they make their way through the course. 
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Sometimes it appears desirable to have two or more senior professionals 
develop the course simultaneously. For a course where modular units 
are being designed to function independently, the anployment of two 
or more developers is a canmendable time-saver. However, in the case 
of a highly structured course where sequential learning is important, 
such as literacy or a second language, it raises many problems. But 
let's assume it's necessary in our case. Here are some of the problems 
and a few solutions: 

1. If chaos is to be avoided, one of these developers 
must take the main responsibility and make the 
final decisions. In fact, one person should guide 
the preliminary planning and design, so there will 
be unity of purpose and approach, 

2. When a canpetent second developer is employed, the 
only way to profit fully from this talent and fin- 
ancial outlay is to give the person freedm to be 
creative. So the two developers need to "get on the 
same wave-length" and then develop their respective 
parts of the course - probably dealing with different 
levels or blocks of the curriculun. They have a 
better chance of getting on the same wave-length if 
they're somevdiere within the same wave-band to begin 
with. This might be ensured by having the chief 
developer select the second one, exploring the 
compatibility of their ideas and convictions. 

3. By allotting blocks or levels of content, the chief 
developer can facilitate some continuity and coher- 
ence in the student materials - at least within each 
block or level. This is achieved partly by making 
reference to previous examples and concepts, as is 
done in the spiral curriculum approach, where a con- 
cept is dealt with several times on successively 
more sophisticated planes. After an orientation 
period, during which the chief developer supervises 
the assistant's work very closely, the assistant 

must be left free to proceed more or less independently, 
supervision occurring mainly on completion of each 
lesson or unit. But how can the developer of level 2 
achieve continuity and coherence by referring to 
examples not yet created by the other developer? And 
let's remember that the first unit of the second level 
must be created at a time when the first level is barely 
started. Here, a certain lack of continuity will pro- 
bably have to be tolerated for the field test version. 
When resulting revisions are being made, the chief 
developer (perhaps, by that stage, being the sole 
developer) can bring the second- level units into 
continuity and coherence with the first level, which 
we will assume he elected to produce himself. 
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4. Developnental testing also presents a problem. The 
only students mAio can properly test the early units 
of level 2 are those v4io have already covered level 
1, vtfiich does not yet exist. Sane conqpromise has 
to be worked out here, and it won't be entirely 
satisfactory. 

But let's adnit that two or more developers are sometimes considered 
necessary in producing this kind of course, and that this arrangement 
has been used with success. The main point is that one of them should 
be in charge of the project, and that this fact should be understood 
and accepted by all concerned. 

The last question is, "How much will the development cost?" This 
depends on the time it takes, the nuiiber of professional and supporting 
staff members involved, the media selected, the processes required to 
produce the "software" for those media, the size and geographic locations 
of the field- test samples, and the woik required for the kind of feed- 
back analysis decided upon. It cannot be e3q)ected that eventual sales 
will repay the cost of development, and this is v*iy a full-scale course 
development project has to be sponsored by some organization which can 
cornnit the needed funds. It is also, probably, the main reason why 
conmercial piilishers have not been able to satisfy all the specific 
needs for training materials, and all the needs to develop innovative 
materials to fit the changing times. 

It is safe to assume that the development process will require 
two, three, or perhaps four years. Therefore, the adninistrator may 
be spared the necessity of stating a total budgetary amount, except 
as a tentative projection, and may be able to arrange the allotment 
of annual amounts for the work in hand. 

As we said at the beginning, course development requires profes- 
sional e3q)ertise, clerical and technical suRwrt, money, and TIME. 
Money, as well as time, is an unknown quantity. 



CHAPTER VIII 
PLANNING AUDIO VISUAL SOFTWARE 

Ross IngToville* 



• Head, Audio Visual Technology Ihit 
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EVERYONE ACTED. BUT INSTEAD OF HELPING TO SOLVE PROBLEMS ... 
THIS RUSH INTO AUDIO VISUAL HAS CREATED THEM. PEOPLE NOW HAVE 
ALL THIS HARDWARE BUT ... IT ISN'T ENOUGH. 
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THEY N^ED THE SOFTWARE ... SUPPORTIVE MATERIALS, THE FILMS, TAPES, 
SLIDES. THESE MATERIALS ARE USUALLY GROUPED UNDER THE HEADING 
OF SOFTWARE ON TODAY'S MARKET BUT IT NEVER SEEMS TO BE QUITE 
RELEVANT. 



PRODUCING PiE SOFTWARE LOCALLY, WITH THE AID OF KITS AND TRAINING 
COURSES, DOESN'T WORK TOO WELL EITHER. 




BECAUSE MOST EDUCATORS AND TRAINERS JUST DON'T HAVE THE TIME. 
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AND WHEN THEY DO HAVE THE TIME, THE SHOWS THEY PRODUCE MAY 
NOT FULLY CAPTURE THE STUDENT'S ATTENTION. 




ON THE OTHER HAND, THE MATERIAL PRODUCED BY AV SPECIALISTS MAY 
HAVE A MORE PROFESSIONAL LOOK, IT MAY HOLD THEIR INTEREST — BUT 




IT MAY NOT TEACH. 
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WHAT IS NEEDED IS A PLANNING SYSTEM THAT WOULD ASSURE AN 
EFFICIENT BLEND OF THE INSTRUCTIONAL AND AUDIO VISUAL SKILLS 
IN THE DESIGN AND PRODUCTION OF THE SOFTWARE. 



) 















FIRST ... IT MUST BE AN INTEGRAL PART OF THE PROGRAM ... 
NOT JUST A (NICE TO HAVE) "ADD-ON". 




SECOND, IT MUST TEACH ... AND WE MUST KNOW THAT IT TEACHES ... 
THEREFORE IT MUST BE BASED ON TESTABLE OBJECTIVES. 



&lffliffl|-ff?lff7l11^H ^ 
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THIRD, IT MUST BE RELEVANT TO THE STUDENT'S NEEDS ... THIS 



MAY ENTAIL LOCALIZING SOME OF THE INSTRUCTIONAL COMPONENTS. 




AND FOURTH ... IT MUST BE APPEALING ... IF WE CAN'T KEEP 
THEIR INTEREST AND CREATE A DESIRE TO LEARN, THE WHOLE 
PROGRAM WILL BE LOST. 
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REQUIRES ONLY 9 STEPS. AND TO KEEP EVERYONE THINKING ALONG 
THE SAHE LINE, LET'S ASSUME THAT WE ARE GOING TO INVESTIGATE 
THE MEDIATION REQUIRED IN A PORTION OF THE MATHEMATICS 
CURRICULUM FOR ADULT REFRESHER TRAINING. 




FIRST, THE EDUCATIONAL OR TRAINING SPECIALISTS MAKE AN "IN-DEPTH" 
STUDY OF THE NEEDS OF THE TARGET AUDIENCE TO DISCERN WHICH 
LESSONS NEED MEDIATION. 



LEARNING N^EDS OF 
JARGET AUDIENCE 



^ V'/?y CONTENT 
ttRMINAL CONOmON^ 





1 



SECOND, WE HAVE TO ESTABLISH WHAT ENTRY SKILLS THE STUDENTS 
HAVE WHEN THEY START THE LESSON AND WHAT SKILLS THEY SHOULD 
HAVE WHEN THEY FINISH. 
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LEARNING M8EDS OF 
TARGET AUDIENCE 



ENTRY I CONTENT 
TERMINAL CONOmONi 
i 

f 

STATE OBJECTIVES 




THIRD, ESTABLISH THE INSTRUCTIONAL OBJECTIVES FOR THIS MEDIATED 
SECTION. FOR EXAMPLE, GIVEN ALL PERTINENT DATA, THE STUDENT WILL 
AND SO FORTH. 



LEARNING NllEDS OF 
TABQET AUDIENCE 




ENTRY / CONTENT 
TBikNNAL CQNOmOm 

r 



FOURTH ... WRITE THE SCRIPT FOR THE MEDIATED SECTION. 



IT! 



lilllMltL GJpimTIOMS 



WRITE SCRIPT 



SELECTION OF 
MEDIATION METHOD 




IN THE FIFTH STEP, WE HAVE TO SELECT THE METHOD OF MEDIATION. 
IS MOTION ESSENTIAL? ARE STILLS MORE APPROPRIATE? WHAT FACILITIES 
ARE AVAILABLE TO OUR TARGET AUDIENCE? THIS DECISION GOVERNS 
THE STORYBOARD PROCESS THAT TAKES PLACE IN STEP 6. 
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LEARNING NBEDS OF 
TARGET AUDIENCE 


41 


WRITE SCRIPT 
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ENTRY / CONTENT 
TERMINAL CONOmONS 

i 




SELECTION OF 
MEDIATION METHOD 
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* * 
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f STATE oBJEcnves 




STORY BOARDINQ 


% 




1 













THE SIXTH STEP - THE SCRIPT IS HANDED OVER TO THE AUDIO VISUAL 
SECTION FOR STORYBOARDING. THIS IS THE MAJOR STEP IN THE PLANNING 



PROCESS BECAUSE DECISIONS CAN BE MADE HERE BEFORE THERE IS ANY 
MAJOR COMMITMENT OF FUNDS. STORYBOARDING INVOLVES THE USE OF 




A FORM WITH A SPACE WHERE THE VISUAL INFORMATION CAN BE 
SKETCHED OUT. 




ANOTHER SPACE IS ALLOTTED FOR WRITING THE NARRATION AND LISTING 
SOUND EFFECTS. 
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THE THIRD "REMARKS" SPACE MAY BE ADDED FOR SUCH THINGS AS 
LIGHTING EFFECTS, PROPS, TYPES OF COSTUME, ETC., BUT NOT 
EVERYONE USES IT. 




AND ADDED TO THESE THERE IS AN AREA USED BY THE PRODUCTION 
STAFF TO KEEP TRACK OF THE SHOOTING SCHEDULE. 




BECAUSE THE STORYBOARDING FOR A MOVIE OR TELEVISION PRODUCTION 
IS MORE COMPLEX 
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THAN THE STORYBOARDING FOR A SLIDE STORY OR FILMSTRIP, LET'S 
ASSUME THAT THE DECISION IN STEP 5 WAS TO PRODUCE FILMSTRIP. 



START 



FOR TECHNICAL REASONS, ALL FILMSTRIPS BEGIN WITH THE FIRST 3 
FRAMES LABELLED "START". 




THEN, SIX BLANK FRAMES, 
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FOCUS 



AND THEN A FOCUS FRAME. 



^^^^^^^ 

FOCUS 



so WE MAKE A STORYBOARD FORM FOR EACH ONE AND PIN THEM UP IN 
SEQUENCE. 




THERE IS ONE MORE BLANK SO THAT THE SUBJECT WILL NOT BE 
REVEALED UNTIL YOU ARE READY. 
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THE NEXT FRAME IS THE TITLE FRAME AND THERE IS A STORYBOARD FORM 
FOR IT. WE'LL START THIS FILMSTRIP BY REVIEWING A PREVIOUS 
LESSON WHERE WE USED PERCENTAGE TO CHECK A CHILD'S REPORT CARD. 




THE SKETCHES USED IN THE VISUAL AREA OF THE FORM SHOULD BE np THE 
THUMBNAIL VARIETY BECAUSE, IN ALL PROBABILITY, THEY WILL BE 
CHANGED. THEY SHOULD BE CLEAR ENOUGH TO ESTABLISH THE INTENT BUT 
GREAT DETAIL IS NOT NECESSARY. 




THE NARRATION TO BE USED IS WRITTEN IN THE AUDIO FRAME AND 
EVERY WORD WE INTEND TO SAY MUST BE SHOWN. 
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THERE IS ONE STORYBOARD FORM FOR EACH STEP IN THE REVIEW. 




NOW OUR STORY STARTS, 




AND AGAIN THERE IS ONE 
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STORYBOARD FORM 





FOR EACH INSTRUCTIONAL 




STEP IN THE LESSON. 
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AS OUR STORY DEVELOPS, WE KEEP ADDING 




THE STORYBOARD FORMS TO THE SEQUENCE. EXAMINE EACH STEP 
CAREFULLY. 




IS THE VISUAL NECESSARY FOR THE UNDERSTANDING OF THE LESSON? 
SHOULD IT BE ONE VISUAL, OR SEVERAL? 
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AND WHEN WE REACH THE END OF THE LESSON 





111 i^^l. 

1^" mm li" 




H 



lii lij^ lif itt 

S^Kmi SSS???! SSuRR 

ENO END 



WE HAi/E THE TECHNICAL REQUIREMENT OF 4 BLACK AND 2 END FRAMES. 




NOW, 
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IT IS LEFT ALONE, 




FOR AT LEAST 24 HOURS. 
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AT THIS STAGE, THE AV DEPARTMENT CAN MAKE AN INITIAL ESTIMATE 
ON THE MANHOURS, TECHNICAL COMPETENCIES AND MATERIALS REQUIRED 
TO MEDIATE THIS SCRIPT. 



IDDDDCDCDDnCI 



laDDnannDDD^DDDDnnnDDDC 
inmMnnncciomDnnDnra 



NOW THE EDUCATIONAL OR TRAINING SPECIALISTS ARE CALLED BACK IN 




HHfflHHHHj! 

□□□Lj^nnnri 




□□□cM]DDaa 



ALONG WITH THE SUPERVISORY OR MANAGEMENT PERSONNEL CONCERNED WITH 
THE PRODUCTION. WHEN THIS PROP'ICTION w"»S IN SCRIPT FORM, THERE 
WAS A DANGER IN THAT 



ERIC 



177 




178 




BUT NOW THAT THE VIEWERS CAN SEE BOTH THE VISUALS AND THE 
NARRATION THL PRODUCER'S CONCEPT CAN BE SEEN AS A COMPLETE 
ENTITY. 
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ALTERNATIVES CAN BE DISCUSSED, SEQUENCING EXAMINED AND CHANGES 
rtADE. THE DESIGNER CAN SEE IF THE AV STAFF HAVE CAPTURED THE 
INTENT AND THE AV STAFF CAN EXPLAIN PRODUCTION TECHNIQUES AND 
LIMITATIONS. 



LEARNIHQ NEEDS OF 
TARGET AUDIENCE 

ENTRY I CONTENT 
TERMINAL CONDITION. 



WRITE SCRIPT 



SELECTION OF 
MEDIATION METHOD 



31 STATE OBJECTiyES 





NEXT, BOTH MANAGEMENT AND THE DESIGNER SHOULD REVIEW THE DECISION MAr£ 
IN STEP 5. WAS THE STILL FORMAT JHE RIGHT.CHOICE? WOULD MOTION, IN ANY 
PART OR ALL OF THE iRODUCT\^! , S^MORE SATISFACTORY? V THE DECISIOji) IS 
MADE^TO GO TO MOTION, OR EVEN pUt MOTION, WE MUST RE-STORYBOARD THE SEQUENCE. 




BECAUSE THE PLANNING FOR MOTION BRINGS IN OTHER VARIABLES, SUCH 
AS 
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BUT EVEN THOUGH THE STORYBOARD FOR MOTION IS MORE COMPLEX AND 
TIME CONSUMING, MOTION SHOULD NOT BE AVOIDED IN THE OVERALL 
TRAINING PROGRAM. 




WE'VE HAD 16 ^W EDUCATIONAL FILMS FOR SOME TIME, 



» * 



\' u 




3 



■ V 



AND, DEPENDING ON OUR LOCALE, WE'VE HAD EDUCATIONAL BROADCASTING 
FOR SOME TIME. 
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BUT THE IMPACT OF VIDEO TAPE RECORDING AND SUPER 8 FILM HAS 
OPENED NEW HORIZONS TO THE EDUCATIONAL PLANNER. 




THE VIDEO CASSETTE IS IDEALLY SUITED TO SITUATIONS REQUIRING 
EXTENDED MOTION SEQUENCES AND, ALTHOUGH IT HAS ITS LIMITATIOi 'N 
SOME AREAS, IT IS RAPIDLY BECOMING TK"^ STANDARD TOOL FOR DISTRIBUTION. 
BUT - IF WE ARE GOING TO US^ IT - WE HAVE TO BE CAREFUL ... BECAUSE 
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THROUGH YEARS OF EXPOSURE TO 



THE TELEVISION SCREEN, 



OUR TARGET AUDIENCES 



HAVE BECOME ACCOMPLISHED CRITICS. 



REGARDLESS OF 




OR IF IT'S A VIDEO TAPE TRAINING FILM. 
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THEY EXPECT TO SEE A PROFESSIONAL 



PRODUCTION. 



* 



BECAUSE OF THIS, THE PAST PRACTICE OF USING OFFICE HELP AS 
ACTING TALENT 
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IS FINISHED. THE ACTING TALENT HAS TO BE GOOD. 




THEY HAVE TO BE PROFESSIONAL. 




THERE CAN NOT BE ANY MORE PACKAGES PUT TOGETHER WITH A "THIS'LL 
HAVE TO DO" ATTITUDE. 
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THE EDUCATIONAL SPECIALIST AND THE AUDIO VISUAL DESIGNER HAVE TO 
BE HELD ACCOUNTABLE FOR THEIR EFFORTS AND THIS 




IS GOING TO MEAN STORYBOARDING OR PLANNING Vllffi DETAIL. 




THE AUDIO PORTION OF YOUR PRESENTATION CAN BE ALTERED TO SUIT 
THE VIEWER. 
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NORMAL 
120-140 
WORDS PER MIN. 



A PERSON NORMALLY SPEAKS AT AN APPROXIMATE RATE OF 120 TO 140 WORDS PER 
MINUTE BUT. WITH VERY LITTLE TmINING. THE LISTENER CAN COMPREHEND 
INFORMATION AT RATES UP TO 300 WORDS PER MINUTE. 



NORMAL SPEAKING RATE 

120-140 NOW AT 200 

WORDS PER MIN. WORDS PER MIN. 



THERE ARE ELECTRONIC 'PEECH COMPRESSORS AVAILABLE TODAY THAT WILL SPEED UP 
THE RECORDED NARRATION WITH VERY LITTLE DISTORTION. RESEARCH HAS PROVEN 
THAT PEOPLE CAN COMPREHEND INFORMATION AT A "FASTER THAN NORMAL" RATE. AND. 
RETENTION IS USUALLY BETTER BECAUSE THE LISTENER MUST PAY ATTENTION. 



NORMAL 
120-140 
WORDS PER MIN. 



WHEN WE RETURN TO THE NORMAL RATE ... THE SPtECH SEEMS TO BE ALMOST 
TEDIOUS ... IF WE WERE TO TAKE A "LISTENING COURSE" IN A NEW LANGUAGE 
OR WANTED TO BRUSH UP ON OUR HIGH SCHOOL TRAINING IN ANOTHER LANGUAGE. 
EVEN THE NORMAL RATE OF SPEAKING WOULD SEEM TO BE TOO FAST. 
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BUT IF IT WERE SLOWED DOWN TO HALF THE NORMAL RATE ... WE WOULD 
HAVE A BEHER CHANCE OF GRASPING IT. 



bO 


hiOVil'lO o SC'Ipl 


OF 


would make it niuch ^ 


:-='<^1AL RATE 


easier to follow, 

-» 


(IN ENGLISH) 


HAVING A SCRIPT WOULD MAKE IT MUCH EASIER TO FOLLOW. 


'}■} 


>* 

\]n i'rangais: 




Si j'avais un tcx(<>. 


■ '^■Al RATr 


('(' scrait bcaruoup 




plus facile. 






(IN FRENCH) 


NOW THIS LANGUAGE WOULD BE MUCH EASIER TO FOLLOW 



IF IT WERE ACCOMPANIED BY A SCRIPT. 
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50 kis'pin 0tt otkiM .k€ 

* musituhikatok m/sftk/ 

NORMAL RATE ,ohtMm..k. 



(IN CREE) NOW THIS LANGUAGE WOULD BE MUCK EASIER TO FOLLOW IF 
IT WERE ACCOMPANIED BY A SCRIPT. 





THE PLANNING OF DETAILS FOR THE STILL PRODUCTION. OR MOTION 
SEQUENCES 




OR. THE COMBINATION OF THESE TECHNIQUES 



EXPANDED NORMAL COMPRESSED 
SPEECH SPEECH SPEECH 



AND THE DETAILS OF THE AUDIO JECTION 



LEARNING NEEDS OF 

TARGET AUDIENCE }i WHirt SCRIPI #U 



EHTEY / CONTENT -^^^ SELtCTlON OF gvi 

TERMINAL CONDITION W MEDlATIOhi METHOD %'M 

STATE OBJECTIVES -Vt^'^^"^ T | 




MUST ALL BE DONE NOW BECAUSE THE NEXT STEP IS THE EXPENSIVE ONE. 



TARGET AUDtENCt 



■A 



^'RITE SCRIPT 



PRODUCTION 



ENTRY CONTENT 
TERMINAL CONDITION. 



SfATE OBJECTIVES 



Sfiit :.roN or 

MEDIATION METHOD 



SrC'^f FJOARDING 



THIS STEP INVOLVES THE PRODUCTION OF GRAPHICS, PROPS, THE 
PHOTOGRAPHY, RECORDING, EDITING, PRINTING AND PACKAGING 
OF THE PILOT PROGRAM 
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LEAmtINQ tmOS OF 

m TA/ner mjoemce 




WRITE SCRIPT 


T 


PRODUCXION 


mm EMTPY / CONTENT 




SELECTION OF 






H TERMINAL CONDlTfONS 




UEDiATfOH UETHOO 


f, 


PILOT COURSE -i 






STORY BOJi^RDfNQ 




1 *' 



WHICH WILL BE USED TO TLST AND EVALUATE THE PRODUCTION. 



rnndffcoMTEMT 
xmmiAL coNomoNi 



r^WF OAJECT7VES 



WRITE SCRIPT 



SELECTION OF 
MEDIATION METHOD 



STORY BOARDING 



PRODUCTION 



PILOT COURSE 



IMPLEMENT 



THEN. BASED ON THIS EVALUATION, THE CORRECTIONS ARE MADE AND 
THE FINAL PACKAGE IS ASSEMBLED AND DELIVERED. 
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BUT TODAY THIS PICTURE MUST BE PRESENTED IN A NEW WAY. 
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IF YOU ARE GOING TO KEEP AHEAD OF THE KNOWLEDGE THAT IS 
AVAILABLE TO US TODAY, NEW DEMANDS UILL BE PLACED ON YOUR 
COi«UNICATION SKILLS. THESE DEMANDS MUST BE MET. 



.OU^ IDEAS ^ . ^,^^^,,3 

YOUR CREATIVE AUDIOVISUAL 
DIALOGUE TFCHNQLOGY 



YOUR CREATIVE DIALOGUE ... TODAY'S AV TECHNOLOGY ... 
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TO BECOME A DYNAf«C INSTRUCTIONAL FORCE. 



msmDciioNK 
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I. GENERAL ORIENTATION TO EVALUATION IN PROGRAM DEVELOPMENT 



A. Introduction 



The concept of evaluation introduced in this paper is very general 
involving, as it does, the recognition that whenever a decision or choice 
has been made, there also resides an evaluation. Hew adequate the data 
used for the decision might be questioned; nonetheless , there has been a 
decision. To make the decisions more relisble and valid, the process of 
evaluation should be made as explicit as possible. Evaluators cannot 
state that they will have nothing to do with any decision based on infor- 
mation because they do not approve of using "inadequate data". ffc>pefiilly, 
one effect of this paper will be to provide insight into the conqplexity of 
evaluation and the magnitude of the task. It should not be looked on as 
an afterthought to course/program development. 

Evaluation is not a negative process although to many people it is 
synonomous with criticism. It is not contradictory to other knowledge 
and data gathering methods . Both practical or applied knowledge and 
evaluation-based knowledge are the result of a cunulation of selectively 
retained ideas and approaches i/rtiich remain after others have been weeded 
out by experience. Evaluation is not in c nflict with intuition, prac- 
tical experience, genius, or tradition. Instead, it is supplemental to 
these processes, attenpting to make the selective process of this evolution 
of procedures ard ideas more precise and reapid. Basically, it is a way 
of increasing the exactness of the testing, probing, and selecting process; 
a refining process super-imposed i?x)n the many valuable cunuilations of 
wise practice. (Campbell, 1959) Evaluation's major virtue (if properly 
done) involves providing a relatively systematic method of checking and 
correcting preconceptions and prejudices; in general, keeping all involved 
more honest and less subject to self-deception (the reality testing function) . 

Steele, in her survey of 50 different approaches to program evaluation 
says: "Ideas ^out evaluation are changing. Beliefs about program eval- 
uation plateaued for a few years during the 1950 's and 1960 's v*ien evaluation 
was equated with research methodology to such an extent that sometimes the 
tenns measinrement and evaluation were treated interchangeably. IXxring 
that period, too, evaluation was often limited to determining whether 
content -specific objectives had been achieved. 

'Then the late 1960 's brought an influx of new programs and new 
demands for evaliiation. Established concepts didn't deliver. As a 
result, new xueas about evaluation emerged and new frameworks appeared. 
There's considerable divergence in those ideas. Most of them are still 
in the trial-and- testing stage. Many paths are being ^aken off the 
plateau of the earlier period, but few of those paths are widely accepted. 
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None can be considered the main route. Some e3q)lore evaluation from the 
standpoint of its purpose, some fron the standpoint of need, some from 
the view of organization and system, and some from the interactive elements 
involved. New definitions of evaluation are evolving . . . development of 
theory and practice is not yet sufficient to completely deal with current 
needs. However, the key ideas that are emerging can help you develop 
strategies that will work in your situation." (Steele, 1973, pp. 20-21) 



B. Program Evaluation as Related to Research 



Steele differentiates Program Evaluation (the focus of her monograph) 
from Program Research and Evaluative Research' . The distinctions are not 
clear cut, since each may have an ii^t into program development and each 
may use similar methods and teiminology. However, understanding the 
differences between program evaluation and researdi will provide researdiers 
with a clearer concept of the task before them ^en confronted with pro- 
blems of program evaluation. Grasping this distinction may ward off 
conflicts among the members of project staff (e.g., program managers, 
developers and evaluators) idiich have been diaracteristic of organizations 
seeking to develop complex programs to solve social problems . Researdiers/ 
evaluators in these settings frequently have taken a negative (critical) 
stance towards development similar to outsiders who infom the program 
developers if and when they have come up with aiy thing worth developing. 
This stance produces considerable feelings of threat and accompaning 
Hostility. One reason for the many evaluation frameworks being developed 
ivas the feeling that this was not a feasible method of operating. 

In general, research implies science and in the search for new knowledge, 
program development, and improvement are secondary. The audience of the 
research output is considerably removed from developers and managers , 
usually being aimed at the larger professional community of science and 
research. The interests, methods of operating, and communication modes 
of these people are quite different from those of program managers and 
developers vtfio are trying to cope with veiy practical and imnnediate pro- 
blems of improving programs. Program evaluation is not trying to obtain 
generalizable infoimation or approaches although this may be a side pro- 
duct. The focus is to obtain sound and reli^le information to use within 
the context of a specific program. While it resembles research orientations 
to programs and is not necessarily in conflict, program evaluation is a 
field in its own. Processes and procedures must be developed within its 
own context rather than be borrowed directly from the research aspects. 

'*lhe confusion of evaluation and researdi has led to underdevelopment 
of evaluation procedures in at least five main areas: suiting data 
techniques to evaluative needs, understanding criteria, distinguishing 
between description and judgement, providing useable infoimation, and 
communicating evaluative findings. 
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"First, although evaluation may borrow data treatment methods like 
precise instnmentation and statistical processes from research, they 
must be tested for validity and reliability as tools in evaluation rather 
than automatically transferred and applied as they'd be in research 
situations* Considerable adaptation may be needed. For example, when 
is it best to examine average gain for a whole groiq) and when is it 
better to examine the number of students making specific amounts of gain? 
Is it necessary to rule out chance occurrence at the .05 level, or would 
a .40 or .25 le/el be adequate in evaluation?... The type of data 
recorded on student -teacher interactions might take a different nature 
vAien a program is being evaluated than it does when it will be used to 
test a particular research hypothesis. A good many other examples could 
be given. 

"Although we're becoming aware that new procedures are needed and 
standard research techniques may need to be amended and adapted, this 
is a relatively new field* Some evaluators and researdiers, for example, 
have been suiting the statistical significance level--the level at which 
chance occurrence is ruled out --to the context of the problem for some 
years now. But little has been done to identify the conditions under 
which a .25 or a .10 level may be appropriate in progran evaluation. In 
many instances, evaluation may not require the same degree of stringency 
in data dealing as does research. This may be particularly true when the 
consequences of inaccuracy are relatively small. 

"Second, criteria must receive much more attention in evaluation than 
in research. In researdi, hypotheses serve as the basis for deciding what 
infomation to gather and how it's to be organized to be interpreted. In 
evaluation, criteria- -decisions about what the program will be judged on— 
play the same role. Evaluative criteria aren't always the same ones that 
apply in research, although there's a similarity in the role of criteria 
without dealing with them as important phenomena. The fledging researcher 
has often been so enthralled with the data-handling procedures that he 
hasn't mastered the concept of criteria as it applies in research. So, 
he's unable to carry even the basic concept into evaluation. 

"Although some program criteria presently exist either in a foimal 
or subconscious state, they must be probed for clarity, reality, and the 
extent to which they portray the most valuable things in programs and 
progranming . Programs can appropriately be judged on a wide range of 
criteria, some of which are in conflict with others. Criteria need to 
be ordered in terns of their importance. For example, which program is 
more successful --the one that reaches a large number of people with a 
moderate amount of help to them, or the one that gives great help to a 
very small number? Which program is more successful--the one producing 
efficient results for a short period of time, or the one with only mod- 
erate results with a longer retention time? 
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"Criteria describe what's of value or good. They're debatable. 
Although programmers sometimes wish to avoid the trauma of debate , a 
confrontation on criteria can clear up some of the ambiguities and 
frustrations that plague evaluation. 

"Program personnel and their professional groups are responsible 
for developing, screening, and debating criteria. In doing so, con- 
siderable attention should be given to the views of clientele. Although 
experts knowledgeable in a given field can make reconniendations and 
researchers can help clarify issues and determine the degree to which 
certain criteria are ccnmonly held, it's not the outsider's prerogative 
to establish the criteria for the evaluation c 

"A third major area for development is the greater understanding of 
the difference between description and evaluation. A presentation of 
research type findings about a program usually constitutes a description 
of the program or of its results. Evaluation doesn't really occur unless 
judg«nents are made about such things as whether the processes used were 
suitable and of high quality or whether the results produced were enough 
for the resources expended, or how effective the processes were in pro- 
ducing the results. Research tries hard not to be Judgemental. Evaluation, 
on the other hand, doesn't fulfill its role unless it takes responsibility 
for delivering conclusions that do judge, identify the adequacy of what's 
being described, or show relative advantages or disadvantages of two or 
more alternatives, 

"It's probable that in some situations, particularly viien reporting 
to outsiders, the programmer may only want to describe what happened in 
the program and/or its results* However, in decision making, descriptions 
are useless unless judgen^nts are made and values assigned... 

"A fourth major area for development deals with the inq)ortance of 
being able to use data. It's not enough to just collect accurate data. 
Those data must be accurately and inmediately interpreted within a 
specific context. Interpretation rests with man. Wisdon and experience 
in foming and using criteria, in assessing the limitations of data and 
the potential consequence those limitations pose, and finally in making, 
communicating, and defending judganents are more important skills in 
evaluation than are the skills of da^.a gathering and statistical analysis. 
Scientifically produced data are a valuable input in evaluation, but 
seldom should stand alone as the output from it. 

"Fran one perspective, data-gathering activities are a separate 
function from evaluation. They're essential to it, but can be considered 
as prior and prerequisite to tiie actual act of evaluation. Unfortunately, 
as we've borrowed from research, we've been more apt to focus on the data- 
gathering techniques than we have on the part of research dealing with 
using data* We've hoped that having high quality data would solve our 
problems. We're finding that data in and of themselves give few answers. 
The next step in understanding evaluation and its procedural development 
must deal with processes *.* that are essential in using infomation in 
evaluating programs* 
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'Tifth, our associating evaluation with research sometimes leads to 
unproductive ways o£ coimunicating results. We may be too prone to 
prepare reports in traditional research ways that relegate conclusions 
and implications to the end of the report. The people who should use 
evaluation seldom get to the end of the reports. If they do, they are 
often so confused by academic hedging and research jargon that they 
can't decipher what the evaluation really means in teims of their own 
needs and situation. Currently a variety of ways of inserting evaluation 
findings and conclusions into the bloodstream of the program and commmity 
are being emphasized. 

•Meaningful evaluation has been stunted by failure to recognize that 
program evaluation and evaluative research, although related, aren't the 
same. New that the nature of the differences are clearer, progress should 
be made in developing specific procedures that work well in evaluation." 
(Steele, 1973, pp. 30-35) 



C. The Scope and Definition of Evaluation 



1. A Classification of Evaluation Approaches 



Using Steele's (1973) analysis, but adding the first category, the 
following classification scheme is provided as a framework: 

I. Evaluation of the Siq)er-ordinate (Contextual) Systems 
of a program (the influence of the program on the 
larger network of inter-related social systems). 

II. Evaluation of participants (grading, certification, 
etc.). 

III. Evaluation of personnel (performance review, skills 
surveys, job satisfaction, etc.). 

IV. Evaluation of program. 

A. Evaluation of process (group interaction, 
effectiveness of instructional approach, 
etc.). 

B. Evaluation of results. 

*1. Results --Attainment of Objectives . 

*2. Results --Evaluation of Outcomes and Effects . 

C. Evaluation of programs as functioning, 
producing systems. 

1. Patterns of program systems. 

*a. Evaluation as Input Into Decision Making . 

ERIC * 
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i. Major or general decisions. 

ii. Specific decision situations (selection 
of materials » setting priorities , etc.)* 

*b. Evaluation of Program Parts . 

2. Patterns of program systems as mirrored in 
evaluation. 

*a. Evaluation- -Kinds of Data; Types of Activities . 
*b. Evaluation Processes , 
i. General processes, 
ii. Specifically applied processes. 



* Indicates the six categories Steele used as major 
groupings for the organization of her monograph 
(from Steele, 1973, Table C-1, p. 238). 

As mentioned, Steele surveyed 50 strategies of evaluation and 
classified than into the six categories indicated by * ebave. 

This provides one way of categorizing current approaches to evaluation. 
Althoygh several of the approaches could have been placed in more than one 
category, she groiQ)ed them according to their primary focus. 

The various aj^roaches are not substitutes for each other. Few, if 
any, are in conflict since many can be used in coni)inations of two or 
more. Even the ai^roadies within a category are not different ways of 
doing the same thing and can be used in various cooi^inations, at various 
times, for various purposes. 

The model, **Evaluation of Program Parts", (Category *C.l.b.* above) 
can be used in conjunction with ^'Evaluation as input into decision- 
making" (Category »C.l.a.*) with no difficulty, since the latter deals 
with qualities or stages of progranming \Aiere as the foxmer deals with 
actual parts of that program. An iq;>proach selected from Category 'C.l.a. ' 
would establish the program sequence of evaluate, develop, re-evaluate, etc. 
An approach selected from C.l.b. would indicate idiat element would be 
evaluated. 

The approaches in "Evaluation - kinds of data" (Category •C.2.aO 
emidiasize parts much as those in 'C.l.b.' but refer to kinds of data 
used rather than program parts directly. Some of the approaches in 
the "Evaluation processes" (Cat^ory 'C.2.b-') deal with basic processes 
used regardless of the aspect and the program being evaluated. Appraisal 
processes, which em^iasize hunan judgement, and approaches that emphasize 
use of systematically collected data are not alternatives to each other 
and should be used together. Other approaches in this category deal with 
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specific applications of process; stressing continuous evaluation and 
using either appraisal or systematic data processing or both to assist 
program development. 

The most highly developed approaches are in category 'B.l.', 
••Results - Attainment of Objectives". Much more attention has been 
given to the details reJned to evaluating results using attainment of 
objectives as the basic criteria of success than to any of the other 
foims of evaluation. Most of the approaches in category 'B.2.' try to 
define and describe the types of outcomes and effects that make up 
program results, not restricting their focus to attainment of objectives. 
Methods in categories •B.l.' and •B.Z.* can be subsuned within the 
approaches in categories •C.l.a.', •C.l.b.', and •C.Z.a.*. 

From this brief discussion of how the various categories of evalua- 
tion relate to each other, one readily gains the impression that evalua- 
tion is far from the "cook book*' specification stage and is still more 
art than science. However, with Steele's pioneering effort, the first 
step has been made in a science of evaluation systems usage. 



2. A Survey of the Various Problems and Evaluation Needs in Program 
Development " ' ' 



Steele provides a cross-referencing guide for people involved in 
developing and managing programs to assist them in using the myriad 
approaches to program evaluation. In any complex program, more than 
one evaluaticm framework will be needed since different approaches 
accomplish different things. Table 3 (Steele, 1973, p. 57) lists 
these general needs: (1) frameworks that help make major decisions 
and overall prograin management; (2) frameworks for examining the 
impact and larger results of programs; (3) frameworks that guide the 
organization and use of program components; (4) frameworks for viewing 
the program through the eyes of the participant; (5) frameworks that 
guide evaluative processes; (6) frameworks for examining the results 
of intervention (e.g., instruction); and (7) framewor^vS that serve 
specific purposes. 

Table 4 (Steele, 1973, pp. 61-74) lists problems, with their 

relevant evaluation approaches, under three general headings: Problems 

in program management. Problems in program development and Problems in 
program evaluation. 

Problems in Program Management involve such things as: 

- Guiding a program that involves several staff members 
with roles dependent m prior work by others. 

- Uhderstanding the kind of decisions involved in pro- 
grammirg. 



ERLC 



211 



• Getting the most out of limited resources. 

• Deciding whether ideas for programs are good. 

- Supervising activities of several para-professionals. 

- Securing additional funding. 

- Developing long-range goals and mission. 
" Reviewing long-range goals. 

- Improving teamwork within your unit. 

• lAiderstanding the stages of programming. 

- lAiderstanding relationship of evaluation to decision 
making. 

- Defining program efficiency. 

• Settling disputes when two or more methods or plans are 
advocated for doing the same thing. i 

• Designing a management infoimation system for your * 
unit. _J n 

' " ' ' ^* J ) 

Designing accountability strategies. 

Developing an organization and climate that will iii?)rove 
programming . 

Identifying and weighing groips and other pressures that 
must be considered in prqgranming . 

Getting new ideas adopted by the staff. 

Organizing support resources effectively. 

Increasing operation efficiency. 

Cooperating with content experts. 

Using a systems i^roach to organizing programming. 

Developing criteria to guide the work of your unit. 

Considering ii^ut, process, and output relationships 
of various parts of your program. 

Developing a multicourse curricula or a multi- 
activity program. 
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- Increasing the contributions of volunteers. 

- Inqproving everyday operations. 

- Detexmining vdiether programming activities are on 
target in teims of the unit's mission. 

- Using infomation in making program decisions . 

- Assigning resources to staff units. 

- Managing the teaching* learning transaction. 

- Developing a personnel evaluation system. 

- Analyzing weaknesses and problems in the operation 
of the unit. 

- Working with an advisory coimittee. 

* Improving use of time and other resources. 

* Developing budgets. 

- Guiding and controlling the total programning 
process . 

- Helping staff use evaluation as a management tool . 

Problems in Program Development involve sudi things as: 

- Setting priorities. 

- Choosing among program possibilities. 

- Deteimining the kind of results you are aiming for 
as you design the program. 

- Developing objectives. 

- Deteimining the right level of objectives. 

- Selecting content and focus. 

- Identifying vAiat participants see as appropriate 
program activities. 

- Examining factors affecting participation and learning. 

- Developing a project or program plan. 

er|c '^'^^ 
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Developing a pilot project, 

Detem5jiing hew you can use scarce resources most 
effectively in the program. 

Making sure that all of the elements of the program 
are £q)propriate. 

Understanding hcM various asp^ts of the program fit 
together. 

Working effectively with others in the program. 

Choosing among resources, books, audio-visual packages, 
etc. 

Preparing instructional plans, 

Deteimining Aether a program is progressing toward 
its goals. 

Finding out \diy a program is not producing as much as 
you e^qpect it to. 

Finding out how well your first progranming efforts are 
achieving. 

Clarifying vrfiat kind of results to expect from a new 
program, 

Understanding how program CGn^xments interact to produce 
results , 

Testing a particular instnictional sequence to see idiat 
it is achieving. 

Selecting the most effective method for a particular 
task. 

Deteimining the effect of a prpgram on a commmity. 

Identifying vtfiether you are reaching the right clientele. 

Revising a pilot program and developing a guide for 
others to follow. 

Getting other people's ideas on how your program can 
be improved. 

Improving on-the-spot judgements as a prpgram progresses. 
Examining the results of programs. 
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- Identifying key elements contributing to the success 
or failuie of a program. . . 

- Preparing a program report. 

- Deciding i^iat to emfdiasize in a report prepared for 
administrators and inf luentials . 

Problems in Program Evaluation involve such things as: 

- Inqproving staff attitude toward evaluation. 

- Distinguishing between describing and evaluating 
program results. 

- Deteimining results when objectives were poorly stated 
or changed duri:-g the program. 

- Developing criteria for judging the program. 

- Dealing with several criteria at one time. 

- Forming judgements about programs. 

- Involving lay people in evaluation. 

- Organizing a comprehensive program review, 
accreditation team, etc. 

- Preparing for an ace oditation team, monitoring team, 
or comprehensive rev. w. 

- Deteimining what parts of the program to include in 
your evaluation. 

- Identifying appropriate data bases and sources of data. 

- Selecting techniques to carry out specific evaluation 
tasks. 

- Categorizing kinds of judgements that need to be made 
about a program. 

- Deciding whether to consult an evaluation expert. 

- Deteimining vrtiether an extensive researdi effort is 
appropriate . 

- Handling a good deal of unrelated data and making sense 
out of how it fits into patterns. 

- Setting perfoimance standards. 
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- Determining the kind of data needed. 

- Establishing or evaluating sanpling procedures. 

- Establishing or evaluating measurement procedures. 

- Examining assumptions related to data. 

- Identifying flaws and limitations in data as a basis 
for assessing the probable accuracy of the data. 

- Developing evaluation reports. 

- Getting evaluation findings used. 

- Evaluating evaluation. 

For each of these problen:s Steele lists suggested evaluation approaches 
which, in her judgep»ent, will oe useful. In most cases more than one 
approach is listed for each problem and, of course, any given approach will 
be useful in several proble,.is. However, as is emphasized throughout the 
monograph, there are no guarantees of solutions. Considerable ingenuity 
and skill will be required to solve these problems but one does not have 
to "reinvent the wheel" since there exists a nunber of possible appzxMches 
for a solution. 



3' Differences in Viewpoints and Needs Regarding "Evaluative Infbnnation" 

Steele points out that a major source of confusion regarding evaluation 
involves the variety of people who want "evaluative infoimation". Not only 
do various people desire this infoimation but the infoimation th^r want is 
quite different since they have different interests, needs or tasks. She 
lists six types of people vtfio desire differeht types of evaluative infor- 
mation: Social policy makers. Civic leaders and other influentials , 
Adninistrators of funding agencies. Local adninistrators. Program personnel 
and Clients. *^ 

Social Policy Makers (legislators vtfio introduce and promote legis- 
lation, politicians, leaders of causes) desire conclusive data to defend 
or siq)poTt policy and funding decisions. 

Ciyic Leaders and Other Influentials . because of their foimal or 
infoimai leadership status influence participants and supporters of a 
program. 

Administrators of Funding Agencies tend to be "in the middle" since 
the organization or government branch to which they are accountable, 
require data to siqjport allocations of funds. 
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Local Adndnistrators of Programs need evaluative information to meet 
the requests of funding agencies^ as a base on vdiich to make decisions on 
budget and as a guide for staff when making basic program decisions. They 
use program evaluation as a base for providing administrative leadership. 

Program Personnel (developers, implementers , etc.) need evaluative 
infomation to assist them to develop, test and redevelop their programs 
as well as infoimation on the larger impact of the program. 

Clients are those whom the programs are intended to benefit in some 
manner. 

Since the amount of a program budget alloted to evaluation is usually 
not large, and since evaluation is a complex a tivity, not all of these 
people will receive the evaluative data they feel they deserve or require. 

'\>ne way of redjjcing uneasiness, if not frustration, is clearly 
understanding the various needs involved and the amount and kind of 
activity needed to satisfy them. Then, better dioices can be made 
about how the program unit will invest its evaluative resources . 

'Most models, frameworks, or approaches to evaluation have been 
designed for a specific type of situation. Such frsmieworks seldom 
handle the variety of other needs and evaluative responsibilities of 
local progranming personnel. For example, a model designed to provide 
the social policy maker with sound generalizable data may have little 
value in helping the programmer guide and improve a program that is in- 
process, and vice versa. The kind of evaluative information provided 
by a particular evaluation approach needs to be examined in terms of its 
value compared with other evaluative information provided by other 
approaches. 

"Not only must the six groi;?)S clarify and better coraramicate their 
needs and program administrators take a more active role in trying to 
establish balance among needs, but all six groiq)s must take more res- 
ponsibility for developing evaluation models." (Steele, 1973, pp. 8-9) 

The types of programs considered in this paper are quite complex 
involving many aspects internally and many connections externally. If 
a local program is one of several similar efforts carried out in different 
parts of the country, then the evaluative needs of the "head office" may 
not be the same as for any given local program and may even hinder local 
evaluation by taking up time, manpower and other scarce resources. Thus 
there is need to make the evaluation policies and procedures at different 
levels clear and consistent. This calls for a guide to systenatize 
evaluation systems (Steele, 1973, pp. 9-10). The dependence on funding 
agencies usually complicates evaluation procedures since the funding agency 
frequently has no c^ear idea of what it wants. Thus the local program 
becomes involved in a frustrating and wasteful process of second guessing. 
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These activities have little value to either the local program or to 
higher administration. "Devising an evaluation syston that's efficient 
and effective for all concerned in a mult^ -location, multi-level admin- 
istrative network has different dimensions than a system developed to 
work effectively for the local program." (Steele, 1973, p. 11) 

Programs which address thonselves to social problans, refom and 
seek to produce social inventions (Conger, 1973) are frequently involved 
in conflictual situations posing additional need for evaluative data and 
making the evaluation process more anotionally laden than it is ordinarily. 
Conflicts involve reconciling or at least coping with opposing value 
orientations. "Evaluation can be an important tool in conflict situations. 
However, it must be a kind of evaluation that can deal with value questions. 
As of yet this is a relatively undeveloped area in evaluation theory and 
in the systematic practice of evaluation. The scientist is trained to 
avoid valuing. Thus, his evaluation models may fail to adequately consider 
the process of dealing with values in evaluation." (Steele, 1973, p. 13) 

Such programs will usually have somewhat strained relations with more 
established programs and agencies. The clientele of the new program fre- 
quently are drop outs or rejects from the established agencies and insti- 
tutions and thus the new program needs, more than ever, solid evaluative 
data. However, this loads the program evaluation process with considerable 
"emotional charge". 

Since the approadi to evaluation described in this paper is relatively 
recent there has not been sufficient time to work through many of the 
problems of program evaluation. In this way, the program and the program 
evaluation system have to be developed in a 'Tjoot strap" style of operation. 
"When a program lacks past operating experience, it has a particular need 
for evaluation. For exar^le, evaluation can help programmers build an 
understanding of their efforts that might otherwise take years to identify 
through ordinary activities. Evaluation can r:apidly increase the infor- 
mation available on programming and reduce the need for trial-and-error 
activity. It can be used to build an image that ordinarily comes slowly 
through increased contact with the program by and through the participating 
clientele. When program personnel are alert and strong enough to prevent 
being shackled with old approaches to evalur.tion, lack of past history 
provides an opportunity to build evaluatioTi stiategies that meet their 
unique situation. However, making use of this freedom requires strength, 
perseverence, and the ability to develop .".ogical rationale for the 
evaluative activities used." (Steele, 1973, p. 14) 

Systems analysis has made a major input into the evaluation of pro- 
grams mainly because these programs are complex systems (Romiszowski, 
1973; Stowe, 1973). The "C^n University" of England has made extensive 
use of systems analysis in their development and evaluation activities. 
However, Lewis (1971c) has pointed out that very high powered network 
planning methods (e.g., PERT - Program Evaluation and Review Technique - 
and Critical Path Analysis) are not transferable entirely from engineering 
because too many variables can not be assessed precisely enou^ to allow 
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accurate time estimates, etc. However, for providing the "Big Picture" 
(activity structure, see Annex I) they are very useful approaches. Tt 
can not be assuned "...that the programmer should always be able to 
identify in advance what results his program will produce and, like the 
factory assembly line, produce those results come what may ...[Nor can it 
be] assumed that the programmer should by one means or another, be able 
to control the participants and other human environmental factors in the 
same way that man can control machines. Many programs. • .are producing 
greater and often more important side effects than the specific objectives 
of the program. One way of looking at the finding of important side effects 
is that the wrong objectives were set- -the programmer couldn't diagnose 
what the real needs were. However, another view is that the exact effects 
of a given program input aren't readily predictable. It may have influence 
(helpful or hamful) far beyond its designed intent. Evaluation should 
consider those results as well as how the program accomplished what it 
said it was going to do." (Steele, 1973, p. 16) 



4. Scope of Evaluation 

The general scope of evaluation can be indicated by enumerating four 
facets of evaluation processes: the topic areas where decisions are made; 
the type of data base used in the decision; the source of the data for the 
decision, and the time of the decision. 

Decisions are made regarding policy, level of intervention, type of 
intervention, areas and agencies of intervention, dissonination practices, 
organizational structure, methods and systens of evaluation and measurement, 
theory and models, training methods, student progress, instructor charac- 
teristics (ability, training, personality, style), course/ curriculum content 
and so on. 

The type of data base on which these decisions are based can be 
divided into: 

- data from the implanenters of interventions including such 
things as objectives of agencies, social pressures, personal 
values and prejudices of policy makers, professional judge- 
ments and opinions, priorities and standards, etc., and 

- data from the people receiving the intervention vdiidi include 
such things as attitude and personality measurement, statis- 
tical results (descriptive statistics, difference tests, 
measures of association, etc.), standardized tests, skill 
levels of students, student anployment rates and incomes, 
demographic statistics on students and potential students 
and so on. 
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The source of the data is similar to the data base but involves more 
^tail as to the people on which the data is gathered, e.g., politicians, 
funders, superordinate support organizations, conmunities, user agencies, 
present and potential students or clients, outside professionals and 
consultants, outside evaluators, project personnel (management, development, 
training, evaluation, and support staff) and sometimes even logical and 
rational considerations are used as sources of data for decisions. 

Finally the time of the decision can range from very early policy 
decisions through the various stages of development of a program to 
decisions regarding where to implement programs. All of these aspects 
of the decision/evaluation process need to be considered in foimulating 
a ccmplete evalu£.tion system. 

Given this broad scope it is apparent that the tem "evaluation" no 
longer refers to a well defined and explicit activity. Rather, it is a 
generalized process. According to Steele, "The two most prevalent ways 
ot looking at evaluation as a generalizable process are: 

1. As a process of forming judgements about programs 
using criteria or standards of comparisons and 
descriptions of what occurred and resulted in the 
program. 

2. As a process of using infoimation in comparing 
alternatives in reaching program decisions. 

The first concept places the emphasis on judging and foiming conclusions 
about program activities and program results. These conclusions are then 
fed into decisions about further jnd futu^^e activity. The second concept 
emphasizes identifying alternatives and then using evaluation to help 
choose among those alternatives... There's a good deal of similarity 
between the two concepts of evaluation... Both are so new that little 
attention has yet been given to how they relate to each other--how 
judgement and decision making function in relation to each other. Much 
more exploration of the linkages between the two is needed." (Steele, 

Program evaluation is broader than evaluating particular instruction 
or training effects on an individual. Part of program evaluation involves 
instructional evaluation, to be sure, but it deals with additional aspects 
of the program such as the effects of different combinations or sequences 
of training, or the total impact of participating in a program on an 
individual and his social environment. It also is concerned with such 
things as the relation between instructional or program units and agency 
goals, etc. "Even though good instructional evaluation provides essential 
infomation for evaluating a program, it does not provide all of the 
information that is necessary even in programs where instruction makes up 
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the total of the program. It is possible to do a good job of instructional 
evaluation without ever really doing program evaluation/' (Steele, 1973, 
pp. 29-30) 

Program evaluation is also more than examining the attainment of 
objectives although, again, it includes this as one aspect. Indeed, 
restricting the evaluation of a program to objectives may ignore more 
important results and outcomes • To focus on the results of the program 
means that you look at attainment of objectives, to be sure, but recognize 
that most progranming situations produce unanticipated results and some- 
times the values or harms of these are more linportant than the objectives 
per se . If the objectives stated for a program are not germane or are 
too narrow then important outcomes will be overlooked if the focus is 
too narrowly on attainment of objectives, '^Evaluation which is concerned 
with the overall effectiveness of a program is concerned not only with 
results in teims of behavioral changes in peq)le, but also with the pro- 
portion of the potential clientele that is reached, the balance in types 
of people reached, the extent to which the icsults deal with urgent and 
continual need, and the care with vfliich participant, agency, and societal 
resources are used. Program evaluation is as concerned about the value 
and suitability of the program as it is with whether its purpose is 
accomplished." (Steele, 1973, p. 25) 

Program evaluation is not just descriptive since it gives both 
evidence of results, whether intended or not, and evalijates the results. 

'Tast approaches to result evaluation limited themselves primarily 
to identifying if results did occur and testing whether they probably came 
about as a result of the program. Seldom «iid they try to identify the 
sufficiency of those results... Evaluating results gets into questions 
requiring more infonnation than just whether the program produced results. 
It involves such sticky questions as: ...Are the results important? Do 
they contribute more to the participants and society than if the time 
and other resources had been invested in oHier things? Were they produced 
at a reasonable cost? Are the results sufficient in terns of the over- 
all need? Are they sufficient in the expectations of the participants 
and the amount of time and energy they invested? Is there any evidence 
thit it's realistic to expect a program to produce more results than this 
one has, given the same budget, personnel, and working conditions?... 
Most of these questions can't be answered by traditional evaluation 
approaches. Ways of dealing with these types of questions still must be 
developed." (Steele, 1973, pp. 25-26) 

Finally, program evaluation involves more than evaluating the results 
of a program. Until recently, evaluation was viewed as primarily a sun- 
mative activity in which program results were studied. TTiis involves using 
standard experimental design applied, inappropriately, to complex and dynamic 
intervention systems in the process of developinent. Formative evaluation , 
which influences the program while it is developing, is now seen as equally 
important. Evaluation of the processes of program development and imple- 
mentation (planning, managanent, coordination of resources, iimediate 
feedback of aspects of program intervention resultr to developers, etc.) 
is seen as essential to the overall impact of the program. 
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...the emerging emphasis is on frameworks that consider both nrocess 
and product and, more importantly, the interrelatedness of the two. Program 
evaluation deals with the program as a functioning, producing system. It 
includes results as one key component, but also examines other aspects 
ot the program. Processes and structural ccmiponents such as tea(ier's 
performance, materials, and facilities are examined in relation to how 
they can be more effective in generating a viable, valuable program, 
nie emerging models differ in how they define the parts of the functioning 
system.. Program evaluation serves both during the actual program ooer- 

^ retrospectively analyzing how the particular level of results 
was producai. At a broader level, it expands our knowledge of what makes 
up successful programs under specific environmental conditions fback- 
grounds of students, extent of budgetary support, social pressures). Such 
^eLrLysr^iy-^r^'^^^ ^° controversial programs." 

The broadened scope of evaluation and its essential function in 
management and decision making has meant an increased emphasis on the 

°^ evaluation to people. "Evaluation as input into decision 
making emphasizes the need of interface and interaction. Evaluation for 
program improvement recognizes that those who must make the improvements 
must be actively involved in the evaluation. Recognition of close relat- 
ionships between evaluation and politics and policy emphasizes the hunan 
element. The growing awareness of criteria and judgement's central place 
in evaluation emphasizes that those criteria come from people and that 
judgement is made by people. Even in tenns of data, the need for involving 
a variety of people in interpreting data so that a more complete picture 
can be secured is becoming more apparent. Fbr a few years we tried to take 
evaluation out of the people realm, hoping to increase its objectivity. 
Lately, however, there's a growing return to active involvement of many 
minds as conclusions are fomulated and decisions made. We need ways of 
increasing objectivity while relating closely to the people involved 
during the evaluation process." (Steele, 1973, p. 37) 



5. An "Evaluation Space" 



Figure 1 cOTSitutes an attempt to picture the complexity of evaluation. 
Tlus diagram seeks to indicate much of the possible interactions between 
people in various positions and situations with respect to the developer/ 
evaluator of a given program in the process of development. There are 
300 possible cubes to examine in relation to evaluative concerns and, of 
course, each cube represents a variety of sub -interrelations. We beein 
at the most macro level. ^ 

a. Social Context ; This involves the total social/cultural/ 
economic/political context of a program and includes Government institutions 
(national, regional, provincial, local). Educational organizations and 
institutions (at all levels from pre-school to graduate school, technical 
education, public and private) , Information institutions and agencies 
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(libraries, National Film Board, etc.). Entertainment and Cultural 
institutions (music, arts, drama, movies), Bioadcast and Mass Media 
institutions (ccHimercial and public TV and radio, cable systems, print 
media- -newspapers, magazines, books, etc.). Business and Industry organ- 
izations. Legal and Correctional institutions. Transportation and Dis- 
tribution organizations (post office, public and private transportation) , 
Helping agencies (welfare, manpower, Native societies, John Howard), 
Nationality-Cultural-Linguistic organizations. Service organizations 
(Kinsmen, Lions, YM/YWCA, etc.), Medical and Health institutions, and 
Professional organizations and Unions. If this is not exhaustive, one 
can get the idea. All of these may be related in complex ways and are 
in turn related to a given program. 

b. Population : This is similar to the social context but includes 
those individuals who may not be involved in or identified with some type 
of formal social organization. This includes the total population of 
people touched, intentionally or not, by a given program and is of part- 
icular interest to programs which use mass media in their delivery. The 
intentional population -is usually referred to as the "target population" 
for a program. Promotional and recruitment activities concentrate in 
this area although they use parts of the first classification ("Social 
Context") in their activities. Dissemination of programs can also take 
place in this sphere although it is mainly concentrated in the "Social 
Context" area. 

c. Clients : This is the sub-group which results from recruitment 
activities and promotion. They are "processed" in varing degrees in the 
program (e.g., registration, orientation, treatment/training, discharged/ 
placed/certified, followed up, observed, tested, evaluated, etc.). The 
selection process results from needs and wants originating fran development, 
evaluation, social institutions, opportunities, etc. 

d. Organi zational Context : This includes the aspects of the 
organization which contain the program as a sub-unit. Things like 
organizational climate, management style, interpersonal relationships 
among the administrative, support, development, evaluation, technical 
and implementing staff are included in this area. 

e. Program Personnel : This includes the developnent, evaluation 
and implementing personnel and the types of interactions and roles they 
develop as a result of the process of program development. 

In the figure, these 5 areas make up the vertical and horizontal 
axis since they are interactive an*^ interrelated. The third axis arrays 
the 12 stages of project development described in Annex A. 

An ultimate goal for some super-systan evaluation expert would be 
to provide an integrated fomative/summative evaluation of a program 
integrated with progress/achievement evaluation of clients integrated 
with a feedback and client record system for continual self-correction. 
This development would be integrated with a system to assess the impact/ 
effect on the larger aspects of the program context. 
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II. THE EVALUATION SYSTBl OF THE LIFE SKILLS GOURSE: 
AN ILLUSTRATIVE CASE STUDi 



A. Introduction 



The foimative evaluation system eventually developed for the Life 
Skills Course in 1969-1971 (Lamrock, 1971 and Warren, Lamrock and Himsl, 
1971) can serve as an example of a rather primitive system. As a result 
of struggling with the many problems in our attempts to sort out issues 
of sumative vs foimative evaluation, research vs program evaluation plus 
trying to be of some use for program developers, we made seme initial 
attempts to specify a more general system for developing evaluation systems 
discussed in Part IV of this paper. Our research training gave us little 
help at the time except in data gathering and analysis . Had we the 
benefit of Steele's monogrsqph at the time, we would be much further ahead 
in this endeavour. However, since our initial attempt (Lamrock, Snith 
and Warren, 1971) was one of the 50 evaluation models used by Steele to 
perfoim her analysis and since we had no time madiine, we experienced 
most of the confusions and conflicts which are described in Part I. 



B. Brief Description of the Life Skills Course (from Methods for Hunan 
Resource Develoiynent , Training Research and Development Station, 
Prince Albert,Saskatchewan, Ftebruary 1973, pp. 10-13) 



1. The Need for Life Skills Training 



Life Skills training seeks to provide disadvantaged adults with the 
knowledge and skills to demonstrate conq[)etence in himan relations and in 
areas of life responsibilities. Life Skills means problem solving behav- 
iours responsibly and iqipropriately used in the management of personal 
affairs. A sequence of planned experiences helps students iiiq;)l€ment a 
program of personal development in each of the following areas. 

Developing Oneself and Relating to Others. Students identify and 
develop personal strengths and abilities and engage in a balanced degree 
of Self -determinism; they identify, resolve, and obtain help with per- 
sonal problems; they help each other practise new skills. 

Coping with Home and Family Responsibilities . Students identify, 
resolve, and obtain help with family problems; they learn new skills to 
i]q)rove fanily life. 
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Using Leisure Time Purposefully . The students use free time for 
personal development and social benefit and develop existing or new 
interests. 

Exercising Rights and Responsibilities in the Comnunity . The 
students learn about the corahunity so they can use resources effectively 
and for their intended purposes ; they learn what contributions they can 
make for the benefit of self and conmunity. 

Making Responsible Decisions for Work Future , The students learn 
about different occupations and opportunities, and having assessed own 
interests, aptitudes, and abilities, choose a career goal and plan entry 
into it, so that through further training and experience, a more meaning- 
ful work life can be attained. They learn skills of selecting jobs and 
practise interpersonal skills related to getting and holding a job. 

Life Skills education is an activity program. The adult student 
actively seeks knowledge through lectures, panels, symposiims, reading, 
audio-visual aids, Jiscussions, visits and tours. Insight and under- 
standing is gained through feedback techniques, problem-solving groijps, 
task assigmr'jnts, situational tests and simulated activities. Skills 
are gained t>. rough practical e3q)erience, role -playing, drills and dem- 
onstrations. Interests are developed through plant tours, audio-visual 
aids, reading, c^^ativity exercises, role-plsying and group discussions. 
Students gain self-confidence throygh an observable increase in skill 
competency based on performance and feedback on perfomance obtained from 
other students and analysis of video-tspe feedback. 

The Life Skills course relates to other courses in that its content 
and methodologies give students a more realistic insight into their 
abilities, and provide some skills to handle themselves effectively in 
new learning situations. These skills and insists provide them with 
confidence \rfiich peimits them to learn the basic ^education skills and the 
vocational skills needed. The growth in the personal dimension provided 
by the Life Skills e3q)eriences , complements and supports growth on the 
vocational skill level. 

Instead of receiving knowledge passively, the adult student engages 
actively in deriving, collecting, discovering, and utilizing infoimation 
to solve problems « He writes, researches, conqpares, plans, computes, 
observes, thinks, dramatizes, feels, visits, leads, follows. Only time, 
imagination of the instructors and students and practicality limit the 
activities that occur, as the program is life itself. The student's 
participation in personal goal setting and its modification through 
subsequent experience, integrates these activities. 
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2. The Life Skills Lesson 



Each lesson has five stages of activity, and a skill objective/ 
The skill objective dete mines the nature of the activity in each of the 
five stages. The Life Skills course describes these stages as stimulus, 
evocation, objective enquiry/skill practice, application and evaluation. 

In the stimulus , the coach presents the problan. He does this in 
different ways depending on the requirements detemined by the objective: 
in one lesson, he shows a film; in another, he uses a case study; in 
another, a trust exercise. During the stimulus the coach might provoke, 
migiht inform, or question; he aims to stimulate discussion among the 
students. 

In the evocation , the coach encourages the students to express their 
opinions and feelings related to the stimulus. The coadi remains non- 
judgemental assisting students to express their concerns. Students share 
their knowledge about topic, helping each other to clarify the problem 
situation. The coach helps the students to classify the ideas given 
and to define the problan. He helps them formulate fact-finding questions 
for investigation in the next section. 

Li the objective enquiry/skill practice jiiase , the coadi acts as 
a guid-. Students seek related new knowledge to the problem they defined; 
they search for answers to their questions; they practise new skills. 
In lessons of the first two biases of the course, they might study them- 
selves on video, or use check lists to examine their own bdiaviour. In 
lessons of the third phase, in \ihich they study problems related to areas 
of life responsibility, they might study films, books, clippings from 
magazines, or they might go out of the training centre to seek infoimation 
and answers to their questions. 

In the ^plication 0iase of the lesson, the coadi helps the student 
iqpply knowledge and skills to the solution of a problem. The activities 
resemble real life situations whenever possible. The real life situation 
changes as the course develops. In the early parts of the course, the 
here and now situation is the learning group. In mid-course, the home, 
the community, or the job become the focus; students interact in the 
community, invite outsiders in, or plan simulations of real situations. 

In the lesson. Exploring Expectations of Birployers , mployers come 
to the learning groiq) to participate in a dialogue during the evocation 
0iase of the lesson. In the £^lication i^ase, each student seeks in- 
foimation at an employer's place of business. The data becomes the 
subject matter of later lessons, such as Exploring Job Preferences or 
Applying for a Job . 

In the evaluation phase, the students and coach assess how they did 
and how the lesson helped them. In most lessons, the evaluation is done 
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through discussion, analyzing video-tapes or with a check list. In all 
lessons, the coach notes the individual student's need for further practice 
on the skill objective and plans w^s to provide this . 

In sunmary, each lesson specifies a skill objective whidi focuses the 
lesson activities. The coach brings the student to achievanent of the 
skill objective by the use of a five stage lesson plan: in the stimulxis , 
the coach presents the problem situation; in the evocation the student 
reacts to and defines the problem, sometimes foimally, sometimes not; in 
the objective enquiTy/skill practice the student searches out infonnation 

practises new behaviours: in the gg)plication , he lilies knowledge and 
skill to the solution of the problems; and in the evaluation, he assesses 
vAiat was done and how well it was done. 



3. Lesson Implementation 



In the Life Skills course, the coach uses the techniques of behavior 
training to produce behavioral change in the students. TTiis requires a 
precisely stated behavioral objective recognized and accepted by the student. 
It should carry satisfaction of sane sort in its adiievement. The coach, 
or someone else in this behavior oriented learning groiq;), models the be- 
havior if necessary, and the grwp provides siQ>port to the person seeking 
behavioral change; or if necessary, it may impose sanctions. Evidence 
indicates that the students find satisfaction and accomplishment in the 
readily apparent change in behavior and their increased personal effectiveness. 



4. Videotape Recording and Playback 



Feedback from the group and coach are siqjplemtnted by the use of 
videotape recordings. The testimory of the V.T.R. 6i>eaks for itself with 
indisputable evidence. People see and hear themselves as others see them, 
probably for the first time in their lives. The V.T.R. gives added force 
to the feedback vtfiich grc^ menfcers provide each other. 



5. The Conceptual Basis for Life Skills 



The Life Skills Process/Content model integrates three main process 
dimensions and course content. The course content consists of a sanqpling 
of problem situations drawn from life situations; the lesson titles in- 
dicate the scope of the content: Handling Drinking Problems; Handling 
Changes in My Bdiaviour, Looking One's Best, Relating to Others, Depending 
on Others, Managing Money, Applying for a Job. The three process dimensions. 
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use of knowledge, use of groi:p, and use of problem solving behaviours, 
describe the array of skills lAich Life Skills stxidents use in working 
with the lesson content. The book, Readings in Life Skills , contains 
detailed discussions of the conceptual basis of the course. 



C. The Context of Evaluation in the Life Skilxs Course 



1. The FoTwtative/Sunmiative Evaluation Distinction 



Stufflebeam (1968) offered an educational evaluation model whidi 
distinguished between •'product" evaluation and "process" evaluation. 
Scriven (1967) offered a similar distinction between what he labels 
"sumnative" evaluation and "foimative" evaluation. The product-simniative 
phase evaluates the effectiveness of a project after it has run full 
cycle. Evaluation at this point determines the extent to vdiidi the 
program has met its objectives, or what broader results occurred, etc. 
In contrast, the process- foxmative phase provides periodic feedback 
to those responsible for continuous refinanent and development of 
methods, plans and procedures. The overall objective of foimative 
evaluation is to identify and monitor, on a continuous basis, the pot- 
ential shortcomings as well as strengths of a project and feed this 
infoimation bade into the project in a relatively short turn around 
period. 

These two types of evaluation are not mutually exclusive or in 
conflict: foxmative evaluation does not stop before summative evaluation 
starts. Fbxmative evaluation is continuous and should be carried on into 
the sumative evaluation stage and suimative evaluation provides infor- 
mation for redevelopment. Nevertheless, thinking about program development 
in terms of summative and foimative phases can help determine which 
evaluative tedmiques and data collection procedures are most applicable. 

Suimative evaluation requires the level of control more characteristic 
of experimental design in contrast, to foimative evaluation. Thtas under 
formative evaluation, the evaluator monitors the total situation by vising 
tiie most sensicive non-intervening data collection devices and techniqv^s 
obtainable on crucial aspects of the project. Such evaluation is multi- 
variate and does not specify all the important variables before a project 
is initiated. Because program development is continuous, foimative 
evaluation aims at integrating a niinber of studies employing different 
methods of data collection in a total evaluative stuc^. Each of the 
studies may have special methodological problans but the flaws rarely 
coincide in all of them and conclusions can be reached by summarizing the 
results from each. 
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In general, evaluators have overlooked the opportunity to have real 
impact un the directions of educational and social change by limiting 
their evaluation to final (summative) assessment and neglecting foimative 
evaluation. Certainly summtive evaluation is necessary, but it is for- 
mative evaluation that identifies the need for revisions when the oppor- 
tunities for revision still ex?st. It seans better to extract what we 
can while projects are still fluid than to wait for more definitive 
findings which too often cannot be used to iirplement changes. 

Foimative/summative evaluation integrates different methods into a 
total evaluative jq^proach: direct observations create a global picture; 
surveys and standardized tests supply objective data to identify indiv- 
idual and group differences; case studies offer a connected sequence of 
events to help dete 'e and explain individual and group changes; finally, 
the experiment provi^ a controlled test of the effectiveness of irpecific 
variables, and the ove.all research/evaluation design provides for an 
interweaving of the various methods. 



2. An Evaluation Typology 



Figure 2 provides a quick overview of how we eventually saw the new 
(to us at the time) distinctions of fomative/sunmative evaluation as 
they relate to the more familiar progress/achievement approaches used 
with students in educational and training situations. If one makes 
certain arbitrary distinctions (cuts in a continuun) the horizontal 
and vertical distinctions emerge. On the horizontal axis, there are 
two Types of Evaluation ; "Continuous -Ongoing" evaluation involves 
a relatively continuous monitoring process with more or less immediate 
feedback loops built in; "Initial-Terminal" evaluation involves pre- 
post assessments of change. Turning to the vertical axis of Evaluation 
Focus, and making certain arbitrary distinctions, one can distinguish 
eva uation vAiich focuses on students and instructional processes from 
evaluation which focus on the program, module, or product. These 
types of evaluation are not mutually exclusive since evaluation of 
students and instruction is useful for course development. The eval- 
uation focus distinction mainly centers on who is going to use the in- 
formation and for what purpose. 



a. Focus on Students and Instructional Process 



(i) Progress Evaluation monitors knowledge and skill 
acquisition to feedback into the instructional 
process and is primarily done by the student him- 
self, his peers and the instructor (coach) .using 
evaluation devices built into the program. The 
Life Skills Course involves daily evaluation. 
There are scales available for the coach and 
students to use. There is also spontaneous 
feedback within the group, student remarks 
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expressed outside the group, and others. In addition 
each lesson contains a statenient of the objectives 
and proATides an evaluation period specifically set 
aside to assess what was done, ho^ it was done, how 
it could have been done better, whether or not the 
objectives were reached and any other matter of 
concern to the groiqp. About every two weeks each 
student evaluates the courses he is taking and each 
coach evaluates his students. These evaluations are 
used to facilitate student progress and deal with 
potential difficulties. The course contains specific 
lessons which review and evaluate progress and provide 
students and the coach with infoimation on student 
progress. Group ratings are done at various times to 
detect interacti<m patterns in the group and deteimine 
sii)*groups for special skill practice and peer teaching. 

(ii) Adiievement Evaluation assesses stiident adiievement 
regarding a teminal level of knowledge or skill in 
order to make decisions regarding pass, fail, and 
certification. This evaluation is done by coaches 
and supervisors. More detail regarding the progress 
evaluation methods is contained in Curtiss and Warren 
(1973, pp. 199-239). 

Fbcus on Course Development and Refinement 

(i) Foimative Evalu a ticm involves monitoring the course 
process to feed'^ack infoimation to the course dev- 
elopers. The evaluation is done by observers, 
evaluators, developers, instructors and st^rvisors. 
The system of foimative evaluation described in 
this section of the paper provided infoimation on a 
set of course development questions. The evaluation 
methods included observation by evaluation personnel 
and lesson developers, evaluating group interviews, 
a simulated group problem solving test, coach's, 
rating of students on several behaviors, and so on. 
On the basis of the outcomes of this program of 
evaluation, the problem solving skills were defined 
more explicitly and behavioral role and skill training 
were developed to implement in the course and the 
coach training course. (See Warren, Lamrock and Himsl, 
1971) 

(ii) Suimative Evaluation assesses how well the course 
works, vmat does it do? It involves Pre-Post aiAd 
Follow*Up assessments done by evaluative research 
personnel . 
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EVALUATION TYPE 



FOCUS 





Continuous -Ongoing 


Initial -Teminal 


otuoents 
methods 9 of 
instructional 
or facilitation 


Progress Evaluation: 


Adiievement Evaluation: 


uses enroute object- 
ives, 

Monitonng Knowieoge 
acquisition and skil^ 
development to feed 
back into the in- 
structional process. 
Evaluation done by 
student y groi^ mem- 
bers, instructor, 
facilitator, etc. 


uses entering assessment 
and teminal objectives. 

Assess stuaent acnieve 
ment or teminal level 
of knowledge or skill; 
pass-fail; certification. 
Evaluation done by 
instructors, facilitators, 
and supervisors. May use 
standardized tests. 


Module y 
Course , 
Program 


Formative Evaluation: 

Moni ^OT*! Tier ^lip f*n it*c^ 

process to feed back 
into the course dev- 
elopment process. 
Evaluation done by 
observers, evaluators, 
developers, instruc- 
tors, facilitators, 
sijfpervisors , etc. 


Sunmative Evaluation: 

/\9dCdd nuvv wc±± uic 

product module, course 
or program works . What 
does it do? Pre-Post- 
Follow up assessment. 
Evaluation done by 
evaluative researchers. 



Figure 2: An Evaluation Typology 
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3. Areas of Possible Evaluative Studb^ in the Life Skills Course 



Figure 3 provides a schematic representation of the evaluative 
research context that we eventually became aware of in our process 
of *1>oot strap** learning about prc^ram evaluation. Do not assune 
that we in fact did all the glorious evaluation in all the possible 
dcnains of evaluation. We finally becaooe aware o§ the domains as a 
result of doing the evaluation we did do. (Aunack^ 1971^ pp. 16-28) 

a. Social/Cultural Context > A description and analysis of the 
total context in wnioi programs are to occur should be done. Particular 
attenticm is required concerning factors that might help or hinder the 
program (e.g., economic ^ social and political forces). The assessment 
of total programs treats such infoimation as both cause and effect 
variables; that is^ how does the social context affect the program and 
what effects might the program have on its context. 

b. Potential Clients Characteristics and Situations . Within 
the total domain of Social Context variables ^ special attention is 
given to the "target population" and recipient groi^^s. Infoimation is 
required concerning the traditional individual characteristics (age^ 
sex^ ethnic origin^ grade levels etc.) and also the extra-individual 

or sociological aspects (family » neighbouihood groi^, social relations) 
These aspects may also be considered as cause and/or effect . What is 
the effect > for example^ of a rehabilitation program on the total 
connunity yihere the program is ^lied? Furtheimore^ if the program 
is applied to a small ^ isolated and honogeneous comunity vdiat is the 
effect on the comnasiity of removing the best trainees (potential 
ccnnunity leaders) via job placement in larger comnunities? 

c. Entry BridgiM Systems, The location of a trainix^ centre ^ 
the quality/quantity of^ recruiters ^ and the method of introducing 
selectees into a program (registration^ orientation^ etc.) needs to 
be evaluated. These and related problem areas are of particular con- 
cern to evaluation since generalization of results will be intimately 
related to (and circunscribed by) whatever factors give a selective 
ratiiier than representative sample o£ the target population. In addition 
the effectiveness of selection and supportive bridging of clients may 
be reflected in the **no show" and drop-out rate. Bridging systems may 
deserve attention for purposes other than the controlling of unwanted 
influences. They may be treated as major elements to be examined, as 
for exanq[>le a conq>arison of a "traditional entry" approach with an 
"entry plus si^port" approach. Sudi conq)arisons may be of vital concern 
in programs aimed at "disadvantaged" populations. 

d. Organizational* Functioning . The analysis of organizations is 
a specialized area too broad to cover in anything less than a caiq;>lete 
book anfl such books already exist. As a good start, the evaluator mi^t 
read Etzioni's analysis * the "goal model" vs. "system model" (Etzioni, 
1969) . The major point to be made is that programs are inqplemented 



233 



within some type of organizational context and cannot help but be affected 
by that context. Well designed programs can be mangled by malfunctioning 
organizational machinery. The different graixp variables a program eval- 
uator should consider include: group composition > group structure , (work, 
power, communication, and affect structures), group process, an3 ~task 
effectiveness . These are the more commonly accepted dimens ions oT'gfoxxp 
functioning and are dealt with in some detail in most social psydiology 
text books. 



Total Training Environment > The description of foiroal program 
procedures is a relatively simple matter. A more difficult problem occurs 
when a program is actually a complex of stb -programs. A still more difficult 
task is posed for the programmer and evaluator in relation to informal 
(incidental, latent, side-effect) influences. Bnergent social structures 
within the student population, teacher-student bonds C'pets"), and after- 
hours support or training factors may all operate to produce either ben- 
eficial or detrimental effects; in seme instances, sudi effects may even 
exceed the intended program effects. Uhless sudi possibilities are given 
explicit consideration there is no chance to evaluate the magnitude of 
their influence. 



In-program progress and achievement are basic areas to be 
assessed by any evaluator of program effects. Ffere the pre-post assess- 
ment process enters but also the progress of a student may be used in a 
sunmative manner by developing a cumulative record system idiich suimarizes 
the progress reports shewing changes over time as the result of partici- 
pating in the program. Thus a study of ••progress patterns" could be done 
on '"non-starters", "drop outs", "stop outs", "erratic progressors", 
"smooth progressors"^ and "completors". 

Evaluation should also be done on predictors of adiievement . 
To come tp with predictors, the initial assessments must be made as 
close to the start of a program as possible, preferrably before actual 
entry. It is possible that a given intervention may have a differential 
effect on different stib -populations of clients. Unless this is expli- 
citely recognized at the beginning, the pre-program assessment may not 
provide the necessary information for identifying these sub-groups. 

Exit Bridging Systems > With minor variations, the comments 
made in relation to entry bridging systems hold here as well. The 
distinguishing characteristics of "completors" of programs are, of 
course, crucial for the problem of generalizability. Special enqphasis 
may be put on va* ious types of placement assistance. 

g» Post "Program Life Achievement . All too few evaluation projects 
obtain post -program measures of success. Most stop at the point where a 
client leaves the program. The evaluation of more distant effects is 
usually seen as an extension of the basic design for evaluating more 
immediate effects. The difficulty occurs when the different criteria 
for success within a program and beyond one are usually conceived of as 
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Q Social/Cultural 




Figure 3: Areas of Possible Evaluation 
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Study in the Life Skills Course 
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"intervening states" that manifest themselves in either muted or extended 
fashion over time. Thus an increase in academic skills on program may be 
expected to increase the likelihood of a student's enployability or salary 
level after leaving the program (distant effects) . Obviously a readmin- 
istration of tests to measure academic level is relevant for assessing 
post -course effects but irrelevant for a follow up. On the other hand, a 
program aimed at improving a student's social involvement and participation 
may use measures equally appropriate for pre-program, in-program and post- 
program evaluations (e.g., number of meetings attended, number of contacts 
with the opposite sex, voting vs. no-voting). These are problons of cri- 
terion evaluation. Early in the development of an evaluation system, the 
explication of criterion measures is of major importance. In many program 
evaluation att«npts the criterion problem is by-passed with gross, vague 
and stirring propagandistic slogans. By the time it is given serious 
attention toward the end, it can only become subordinate to the program. 
The basic intent of most programs, however, is to affect a certain change 
in behaviour or conditions and to this end one develops what are presuned 
to be relevant programs. It is to the advantage of both program developers 
and evaluators to make as explicit as possible the objectives of the program 
in teims of follow up assessments. 



4. "Zones" of Evaluation of Program Results 



Zone I: Study of Course Effects in the Learning Laboratory 



The training environment provides a comparatively safe setting 
in which to acquire and use new skills and infoimation. Within this 
environment there is an attempt to approadi the "real world" by the use 
of various simulations, role playing and "games" being the primary 
techniques. "Foimative evaluation" discovers whether or not the in- 
structional methods are producing the desired result on course, providing 
infoimation on What works with whom, what additional training the coaches 
(instructors) require, what revisions in the course must be done to 
produce the desired result (e.g., lesson dianges, sequencing changes, added 
lessons, substituted lessons, needed training techniques, etc.) and other 
modifications while the course is going on. "Suranative evaluation" pro- 
vides infoimation on the total impact of the course however modified it 
may have been. 



b. Zone II: Applications in "Real Life" 



Evaluation procedures must also study the course effects outside 
of the learning laboratory conducted vfliile the course is still in process. 
TTiis field work, while difficult and time consuning, is necessiry to check 
on any transfer of skills and knowledge acquired in the laboratory. 
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The ultimate test of the effect of the Life Skills Course 
involves how much application occurs outside the training context. Each 
lesson has an application phase which, especially later in the course, 
involves doing something "outside"* Ihis provides the initial test of 
the effect of the course. If there is no transfer of skills and knowledge 
outside the learning laboratory then the goal of the course has not been 
achieved. If there is no transfer during the course, then there is little 
likelihood that there will be transfer after course completion. Thus, 
data must be gathered from several sources as to how much transfer occurs 
in the various areas and how much occurring on each lesson is appropriate. 



c. Zone III: Post Course Evaluation - Follow Up Study 



No evaluation procedure can be considered adequate if there is 
no foHow up of the effects of the course. Interview and evaluation 
procedures have been developed for the follow evaluation using the 
opinions of course graduates and thus we obtain their opinions of the 
impact of the course after given periods of time (see Cu:tiss and Warren, 
1973, pp. 196-239). If there is a heavy ©nphasis on the transfer of 
skills into the real world throughout the Life Skills Course, then it is 
hoped that the continued use of life skills will follow quite naturally 
and will not be a sudden transition from the safe learning environment 
to the hard realities of life. Thus, the success of transfer while on 
course to the post course transfer should be compared. This study 
involves considerable time and effort but is essential to any evaluation 
process. 



D. Program Development Questions 



As a result of increased experience in program development, we 
became more sophisticated and systematic in our approach. Thus, we 
foimulated several questions with respect to the redevelopment of the 
course. While still at a rather general level, they did help focus 
the data gathering process. When these general questions were for- 
mulated the data gathering methods could be related to the questions 
to see that all areas were covered (see Table 2) . The questions were: 

a. Which of the behavioral changes specified for the 
students actually occur as a result of the course? 

b. What behavioral changes actually occur as a result 
of the course', but are not specified in the 
objectives? 

c. In what respects does the Life Skills Course fail 
to provide the mportunity necessary for the 
students to achieve the specified behavioral 
changes? 
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d. What changes specified for the course are a function 
of time (no training)? 

e. Does academic iqpgrading only result in the same 
behavioral changes as those specified in the Life 
Skills Course combined with the Basic Education 
Course? 

f . Assuming that Life Skills students have expectations 
for a "traditional learning setting", will the pro- 
vision of such a setting result in more efficient 
development of specified behavioral change? 

g. How do the techniques of skill training produce 
specified behavioral changes more efficiently 
than those techniques in the present Life Skills 
Course? 

h. In what ways does the precision required for the 
implementation of the techniques of skill training 
refine the definition of the behavioral objectives 
in the Life Skills Course? 

i. In idiat ways do the written lesson materials fail 

to give adequate guidance to the coadi for effective 
lesson implementation (timings, sequence of lessons, 
clarity of direction, completeness of direction)? 

j . What skills do the coaches lack in order to 

objectively achieve behavioral change in students? 



E. Life Skills Evaluation Methods and Procedures 



The evaluation system developed for the Life Skills Course had as 
its principle objective the development of the course (i.e., it was 
foimative and process oriented). Thus observation procedures, ratings, 
scales, di<?ck lists, role pl^, and simulations were used to assess 
student progress and skill levels so that course developers could 
refine and strengthen the course. 

Several methods for collecting data were developed and a flow 
chart for each method specified. How these various methods fit 
together in an integrated manner has not been made explicit (i.e., 
the evaluation system was not an integrated one). It should be 
pointed out that each evaluation method used to monitor the process 
could also be used in a sunmative manner by plotting changes within 
the course and sumarizing these changes . Not included in this 
description is a system of progress reports using both student and 
coach ratings on several aspects of the course. (See Curtiss and 
Warren, 1973, pp. 199-239 for the various measures developed to assess 
student progress which grew out of the formative evaluation system 
described here.) 
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TABLE 1: Results of Observers' Categorization of Behaviors 
According to tlie Life Skills Process/Content Model 



Dimension 


1 of total 




responses di ■ 20341 


Use of Knowledge 


30.5 


cognitive 


.5 


affective 


3.5 


psychanotor 


26.5 


Use of Learning Group 


38.7 


safe 


21.5 


careful 


12.2 


risky 


5.0 


Problem Solving 


30.8 


not relevant to p/s 


25.8 


problon not dealt with 


4.3 


recognizes problem 


.7 


defines problem 


0 


chooses alternative 


0 


inqplements alternative 


0 


evaliaates alternative 


0 
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The course changes recommended as a result of the £ori.utive evalua- 
tions are discussed in Warren^ Lamrock and Himsl^ 1971. 



1. Observation (see Figure S) 



a. Purpose, To identify behaviors which occur in response to 
specific lessons » coadi and other student behavior, 

b. Criteria , The behaviors must be observable: word, movement, 
facial expression, body attitude, associations with other persons in 

the learning group. The observer does not rate the quality of a behavior; 
he notes only its presence and in the report, offers an interpretation of 
the behavior, 

c. Decisions , The coach may use these records to modify his coaching 
behaviors to bring about desired dianges in his groap. He uses these 
iimnediately. The lesson developer uses the records to sibstitute new 
directions in the written lesson materials or to add new directions if 

he can establish the relevance between the noted behavior and the lesson. 
These observations provide tlie empirical basis for the description of the 
theoretical Life Skills Process/Content model (Readings in Life gcills> 
1973) . These observations will deteimine the retention and modification 
of the model, 

d. Procedure , The evaluation specifications for Intake K called 
for one observer of each training groisp for all training sessions. Due 
to personnel constraints, however, only one group was observed on a reg* 
ular basis . Since the emi^asis was on developing the methodology and 
validating the Life Skills Process/Content Model (Readiiigs in Life Skills, 
1973) , it was felt that this activity would be useful even thoiitglflEe 
data necessary "^or making group comparisons could not be obtained. A 
behavior analysis which reflects behaviors expected in the lesson was 
used as a guide by the observer. He recorded a simnary o£ bdiaviors 
exhibited by eadi groiq> member at the end of each session. 

The observer watched all training sessions for one of the four 
sid>*groups of Intake K for a period of two months. During the lessons 
all the behaviors manifested by the stud^mts were recorded. These 
behaviors were then categorized along three dimensions of the Life 
Skills Process/Content Model, 

e. Resiilts . Table 1 sunnarizes the observers* weekly report 

for the period of November 23, 1970, to December 17, 1970. The observers 
attempted classify each behavior on each of the three dimensions of 
the model. As can be seen in Table 3, 30,51 of the total observations 
were assigiied to the Student Use of l&iokyledge Dimension > and 30 .8% to 
the Problem Solving Dimension . Of the total nutter of observations 
assigned to the latter dimension, 25.81 were assigned to the •Inot 
relevant to problem solving" category. This seemed to indicate the 
inability of the observer to detect problem solving behaviors when 
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TABLE 2: Relationship Between the 
Development Questions and 

V = evaluation procedure 
providt^ information to 
answer development 
Questions 


EVALUATION PROCEDURES 


Observation 


9 

u 
u 


Gr. Develop. 


Lesson Evi^i. 


Training Stqp. 


Rate Student* 


Inf. Ranarks 


I 

•H 
4-> 


P § P Tests 


Sim/Prob/Solv 


DEVELOPMENT QUESTIONS 


(1) 


(2) 


(3) 


(4) 


fsi 


(6) 


,(7) 








a. Which behavioral chances specified 
for the students actually occur as 
a result of the course? 


✓ 




















b. ^' r^ behavioral changes occur as a 
Jit 01 uie course 9 DUt are not 
specified in the objectives? 














/' 






✓ 


c. In what respects does the L/S Course 
laii to pro vice opporruniiy lor stU" 
dents to achieve the specified 
behavioral changes? 














\ 








d. What changes specified for the 
training?) 






















e. Does academic upgrading only result 
in the same behavioral changes as 
those specified in the L/S Course 
combined with the Basic Education? 






















f . Assuming that L/S students have 
expectations for a "traditional 
learning setting," will provision 
of such a setting result in more 
efficient development of specified 
behavioral change? 






















g. Do the techniques of skill training 
prodixte specified behavioral changes 
more efficiently that techniques 
specinecL in l/o: 


✓ 




./ 

tr 








y 








h. In irfiat ways does the precision re- 
quired for implementation of skill 
training refine the definition of 
the behavioral objectives in the 
L/S Course? 














¥^ 


/ 






i. In \rfiat ways do the written lesson 

mat'PT'ialc -fail t'o ffivp aHpfliiat'P 

guidance to the coach for effective 
lesson implementation. 




✓ 


















1. What skills do the coaches lack in 
order to objectively achieve be- 
havioral change in students? 


/ 








/ 




y 
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used by other people. Attention turned to the resolution of this 
difficulty . 

Four observers studied a videoti^ of a Life Skills group in a 
problem solving session, and using the same observation schedule, made 
independent classifications of the behaviors observed. Although the 
different observers- found problem solving behaviors, they could not 
consistently classify them into one of the five broad categories depicted 
on the Model. As a consequence of this infoimation, the Problem Solving 
Dimension, conceptually described as having 5 phases, was further defined 
by more precise behavioral descriptions of the problem solving behaviors 
vihich they subsume and the conceptual development of the course was 
enlarged to account for the apparent fact that the same behavior could 
at one time be used to define a problem, at another time aim at choosing 
a solution, and at still another time be a part of the application. 



2. Coach 'a Diagnosis/Prescription For Behavior Change (see Figure 6) 



a. Purpose . To pemit the coach to specify which skills need 
emphasis for eadi student, and to specify the tedmique to bring about 
the desired behavior. 

b. Criteria. The coach's dissatisfaction with the development 
of a particular skill by a student and his own ability to bring about 
the change. 

c. Decisi ons. The coach modifies his behavior according to the 
need implied by his observations. 

d. ProceAire . After each training session the coadi identified 

a "coach's remecty^ing behavior" needed to bring about a behavioral change 
in the student. He implemented his plan in the next session, after 
which he assessed its effectiveness. The evaluation foim was conroleted 
each day during the first month of training. 

e. Results . The requirements of the evaluation foim used 
identified the coach's inability to diagnose skill failure and to devise 
corrective instructional plans. It also revealed that the coaches often 
lacked the instructional skill of insisting on adherence to a behavioral 
commitment, as a result, modifications of the coach training course placed 
greater emphasis on the diagnosis of skill failure and introduced some 
techniques of behavior modification methodology, e.g., "the pin point 
behavior", "set goal", and "record progress" sequence (see Curtiss and 
Warren, 1973). 
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3. Coach's and Observer's Evaluation of Group Development in I nter- 
personal Relations Skills csee Figure 7^ 



a. Purpose . To evaluate and improve interaction of gro* 
nenibers* 

b. Criteria . The subjective estimate of "group development" on 
specified scales of interpersonal relations skills as defined in the 
scale descriptors. 

c. Decisions. To change coach behaviors, lesson sequence, etc., 
to facilitate group development basic to individual development. 

d» Procedure and Results . This short rating fonn permitted the 
§ e. coach and other observers to quidcly assess the general level 
of ccnpetence and skill of the groi^ as a learning/helping groiq>. How- 
ever, this fom was little used and thus did not generate mudi infonnation. 

It was revised, integrated with other progress assessment mea- 
sures and included in subsequent coach training (see Curtiss and Warren. 
1973, pp. 199-239). 



4» Observation By Lesson Developers (see Figure 8) 



a. Purpose. To allow the course developers to assess the 
effectiveness of the course materials. 

b. Criteria . Coach and student e3q>ressions of satisfaction with 
lesson content and procedures; the developer's satisfaction that in- 
structors and methodologies are sufficiently explicit. 

c. Decisions. To eliminate weak or wrong instructions in the 
lessons. To decide on the relevance of a lecson topic and its sequencing. 

d. Procedure. After each session the observer recorded his 
conments on the quality of direction and general adequacy of the written 
materials. Several course developers observed Intake K. They recorded 
their observations and recommendations and sent the reports to the 
Si^ervisor of Developnent vtfio directed their me in the lesson redevelopment. 

e. Results . It was found that reporting on specifics of the 
written materials (clarity, timings, sequencing, completeness of directions) 
became confused with reporting the coach's expertise in giving the lesson. 
One could identify comments related to the instructional plan itself, 

but often little was left vtfien conments on coadiing were extracted. It 
was recommended that the coach and observers distinguish between conments 
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on coaching and those on the written instructional plan. Also, departures 
from the written plan were to be recorded in more detail so that the 
developers ^<new the reasons: e.g., group interest, special problems with 
the group, preference of the coach, directions not clear, etc. 

The observers' reports indicated that although the lessons provided 
skill objectives, frequently the level of achievement of the skills was 
not specific enough. Also, the reports revealed a need for greater skill 
practice in order to facilitate transfer of the skills learned. As a 
result of this infonnation, the Life Skills lessons now include opportun- 
ities for the students to teach their newly acquired skills, in order to 
facilitate skill training transfer. , uc 



5« Training Supervisor's Sunmaiy of Coach's La ily Evaluation fsee 
Figure 9) 



a. Purpose . To dooment additional infonnation which may result 
from meetings between the training supervisor and coaches and provide a 
method for supervisors to record their evaluations. 

b. Criteria . The comment has an evaluative quality; it refers 
to student progress, quality of lesson materials, or satisfaction with 
the development of the group. 

c. Decisions. Training supervisor can deteimine the effectiveness 
ot the approach of the coach under an array of circunstances. Areas of 
concern to which the Life Skills Course does not address itself can be 
detemined. 

d. Procedures . After each training session the coach met with 
the training supervisor to discuss the progress of the grcjp and these 
comments were recorded along with any other comments the supervisor may 

e. Results . The supervisor's summary of these meetings was 
forwarded to the evaluation section but these suimaries were found to 
contain little infonnation not gained by other methods and thus were 
discontinued. 



6. C oach's Rating of Students (see Figure 10) 

a. Purpose . To doctment individual student progress over time. 

b. Criteria . The presence of certain specified behaviors believed 
to be important indices of individual development. 
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c. Decisions . The coach detemines which behaviors need more 
o^hasis tor each group member. Tlie developer can reccnmend to coaches 
effective coaching behaviors by specifying them in rewritten lessons. 

d. ^^ Procedure. The fom, 'The Rating of Life Skills Students by 
Loaches , allows the coach to docunent progress. The form includes a 
list of over 30 behaviors considered indices of both group and individual 
progress. (See Curtiss and Warren, 1973, pp. 194-239.) 

e. Results . The first completion of this fom was three weeks after 
the beginning of the course (Intake K). The compilation of results per- 
mitted the coaches to compare the standings of individuals within the group. 
TJis report revealed to the coach where the groups needed more training, 
ihe analysis of the results of subsequent foms pemitted the coach to 
assess individual and group progress. Ihe coaches found tl j infomation 
useful in guiding their instructional emphasis . 



7. Recording of Infoimal Remarks (see Figure 11) 



a. Purpose. To capture remarks about training progress which may 
be made in an informal setting and will therefore often escape the more 
structured methods of data collection. 

b. Criteria. The conment must be either spoken or indicated 
clearly in a non-veibal manner; preferably a voluntary remark. 

c. Decisions. To decide on the need for further investigation 
and to assist in assessing other data. 

d. Procedure . Frequently people make evaluative remarks 
regarding group process in an infoimal setting. These remarks are 
'^corded in a diary for subsequent anal/sis . 

e. Results . The procedures were not implemented and therefore 
did not yield any infonnation. 
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8. Rating of the Interview of the Group (see Figure 12) 



a. Purpose . To provide a means by which a skilled interviewer 
can evaluate tne skill of the coach and the progress of the group using 
the whole group's evaluation of these areas. 

Because all four groups of Intake K could not be observed, 
other methods were devised to obtain comparisons of the possible effects 
of coaching styles on group progress as required by development question 
f (see Table 2) . For this purpose an interview of each groim was done 
by the Supervisor of Training for Life Skills. The suggested interview 
guide instructions, together with the foim used for the ratings, served 
as the definition of the dimensions to be assessed. 

b . Criteria . The judged quality of coach performance and grotq? 
development as defined by the rating scale descriptors. 

c. Decisions . The Supervisor of Training made reconinendations 
on any necessary cnanges to strengthen weaknesses and further group 
development and help the coach become more effective. 

d. Procedures . The skilled interviewer was provided with the 
interview guide along with the rating forms, and instructed to devise 
a group interview vrtiich would last no more than 1 1/2 hours and which 
would provide an indepth probe of the designated dimensions. This 
interview was done on the afternoon of December 23, 1970 with two groups. 
In both cases the interview was done in a smooth and probing manner and 
succeeded in covering the required areas thus allowing meaningful ratings 
to be made. Three observers watched the interview from the observation 
room and made notes and did their ratings there. The interviewer 
(Siq)ervisor of Training) , made hi^ ratings after both interviews were 
completed. In addition, each coach watched his group being interviewed 
and also did the ratings although these are not used in the analysis. 

e. Results . The attempt was made to measure the extent to >Aidi 
the two groups were rated differently by the 4 raters on 9 dimensions. 
The statistical test used was the Mann-Whitney U test (a non-parametric 
equivalent of the "t" test; Siegel, 1956) and although the analysis is 
not strictly appropriate, it provided a rough indication of vrtxether or 
not these two grotqjs were rated differently. The hypothesis tested was 
that "teacher style" groiq) and method was equal to or better than the 
"Regular Life Skills" group and method as measured by these ratings. 
This was the case and it was significant at the 10% level or better on 
each dimension. 

Interpretation . The meaning of these results was ambiguous. 
The stucfy gathered data to answer the development question "f" (see 
Table 2) regarding using a "teacher" style of coaching versus the 
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present Life Skills coaching method. Assuning no other difference 
betv^era groi^ and equally competent coaches, the data supported the 
idea of initially providing a didatic style of coaching. This inter- 
pretation was made ambiguous by several complicating factors: two 
coaches of different sex used two styles on two different groiqps. Any 
one of these singly or in conibination could produce the differences. 
In fact, the two groups were initially significantly different cm three 
derived scales of the 16 PF: the group rated higher was also significantly 
less anxious (means of 4.6 vs 5.2, t = 2.15), more mentally healthy (5.6 
vs 4.6, t » 1.85) and showed greater capacity to learn (6.2 vs 5.7, t » 
2.11). This fact alone could produce the rating differences observed 
in this study. In addition, the two styles of coaching did not differ 
so much on directiveness as in type of directiveness. The general im- 
pression was that the female coadi was more subtle in her directiveness, 
vAiereas, the male coach was more direct, overt and obvious. The most 
that could be concluded was that an initial didatic approach to Life 
Skills was not detrimental to group development and was compatible with 
the desirable outcomes described in the foim, "Evaluation Fbim For Group 
Development: Interpersonal Relations Skills". As a result of the above 
infoxmation, it was concluded that the coaching style could become more 
overtly directive, requiring the students to do the activities and use 
the skills in and out of class, but coaches needed to be trained for 
this. The use of the group interview method for assessment has many 
drawbacks from a pure assessment point of view. However, with a skilled 
interviewer, it could be quite useful to provide a supervisor with a 
comparative assessment of groups and coaches and the approach could be 
usefully incorporated into a supervisor training program. 



9. Simulated Group Problem Solving Tes t (see Figure 13) 



a. Purpose. To measure the adequacy of the Life Skills Course 

in developing problem solving skills and to provide a diagnostic technique 
which can be used to develop problem solving skills in students . 

This was an evaluation method developed to assess the compara- 
tive progress of groups. Again, this was done since observation of all 
groups was not possible and some method was needed to measure progress. 
In addition, the technique could be a useful one to incorporate into the 
course format as a periodic progress check. 

b. Criteria . The judged quality of the students* problem solving 
skills as defined by the rating scale descriptors. 

c. Decisions. To pinpoint the behaviours each student needs 
improvement in and to develop a plan of remedying actions for the coach 
to implement. To detemine the progress of students and groups which 
can be used in deciding about further training or recycling in training. 



253 



d. Procedure . On Ihursday P.M., January 7, 1971, each of the 
Life Skills groups were presented with the instructions and case stud/ 
as a test of their individual and group problem solving skills. In 
addition, for comparison, the matched control group was tested on 
Wednesday evening, January 20, 1971, and were told that this test was 
given to the training groups at NewStart to see how well they could 

do without any training. They were then treated like the training groims. 
The situation was described in the instructions handed to the groups and 
read. Any questions they had were answered and yAien the groi;ps were sat- 
isfied that they understood what was required of them, the case stu(fy, 
"Fred", was handed out to them. Each group was V.T.R.'ed and the tapes 
were to be analyzed by two raters . 

e. Results . The fomal analysis of the tapes using the scales 
and developed descriptors was not done to canpletion since all groups 
performed so poorly that virtually no problem solving behaviours were 
exhibited. Infbmal analyses were made of the tapes by various people 
and a meeting was called to view, analyze and develop reme<fying be- 
haviours for coaches to try to make their groups more skillful. As a 
result of this test the training methods in helpful and problem solving 
behaviours were refined and re -conceptualized with lessons added for 
this purpose. Also, the tedinique developed for evaluation was re- 
fined with several improvements in the evaluation instrtment. 



10. Pencil and Paper Tests (see Figure 14) 



a. Purpose . To isolate student characteristics relat-^d to 
progress and to doctment personality/attitude dianges thought to be 
due to the effects of the Life Skills training. 

b. Criteria . The scores students achieve on specified pencil 
and paper tests prior to training and on the completion of the Life 
Skills Course. 

c. Decisions. To decide the test scores which reliably predict 
success for prospective Life Skills students. To identify which dianges 
in attitude, etc., as reflected by test scores, occur as a result of 
the Life Skills Course. 

d. Procedure . A nunter of tests were identified as useful in 
deteimining personality changes which was judged relevant to the objectives 
of the Life Skills Course. Some of these tests were revised for use with 
our population and were adodnistered pre and post course . 

e. Results . A preliminary report exists in Readings in Life Skills 
Cpp. 241-252) . A more detailed report is in preparation, since this 
constitutes a "suimative evaluation", it is not included here. 
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F. Conclusion 



In the description of the evaluation system for the Life Skills 
Course, several shortCOTiings can be seen. The very serious matter of 
transfer outside of the training setting was not addressed (Zone II 
in Figure 4) although follow up was done on the several intakes (Zone 
III) . Another problem has been the lack of coordination in the joint 
evaluation of Life Skills and Basic Education Courses, kgain these 
inadequacies were recognized but not addressed. 

One of the major results of this formative evaluation system, in 
addition to assisting course development, is reflected in this paper: 
that is, developing a general philosophy of evaluation and specific 
procedures for developing evaluation systems (described in sections III 
and IV). This was not an objective of course development but was a 
"spin off" into the areas of "Evaluative Research" (see part I.B.) 
and contribution to the "Science/Art of Evaluation". 
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III. SYSTBI5 FOR DEVELOIMOfT OF PROGRAMS 



Saskatchewan NewStart began to develop , evaluate and integrate two 
interrelated general models: the first involved the 12 stages of devel- 
oping programs in the area of hunan resource development (Annex A) ; and 
the second included generalized systems for develqpment of ;arious 
ccmponencs of projects in these various stages of development. Parts 
of these two iiK)dels have been developed to varying degrees of completeness. 
This section gives a brief sunmary of the systems which have been developed 
to serve as background to a more detailed sumnary of the procedures for 
developing evaluation systems. 



A* Stages of Program Development (See Am.ex A) 



At present » 12 stages are conceived of in program develqpment. 
Tliese are, in nanie: (1) Concept stufy, (2) Exploratoiy development, 
(3) Prototype development, (4) Pilot study, (5) Advanced development, 
(9) Operational Systems development, (10) Demonstration project, 
(11) Disseminati(»i, (12) Installation. See Annex A for the definitions 
of these 12 facets of development. Nunbers 1 througih 7 take place 
primarily in the Development .^raining/Evaliiation setting lAereas, 8 
throu^ 12 deal primarily with events and settings outside . 



B . Relations Between Developmental Stages and the Systems of 
Development 



Annex B provides a picture of seme of the interrelations between 
the developmental stages and project development. This diagram illus- 
trates the conq)lexities of the simultaneous development of the Basic 
Education and Life Skills Courses with their accaq>anying Coach 
Training Courses. This picture is an idealized model of the develop- 
mental process and should not be viewed as a statement of how the two 
illustrative courses were in fact developed. In the diagram, tl^ stages 
of the developmental sequence are listed along the bottom except for 
dissemination vdiidi occurs in an informal and foimal manner throughout 
all the stages of development. Also, as indicated, a preliminary field 
test stage may be conducted prior to the fomalization of the program 
in order to gather information on the workability of the program in 
settings different ^rcm' the one in which it was developed. 

In the diagram (Annex B) illustrating the initial simultaneous 
development of the Life Skills and Basic Education (plus the coach 
training) Courses, these aspects are diagrammed for each of the projects 
and the interrelations and inputs are indicated by the one and two way 
arrcws. In tids diagram, the place where the various annexes are appro- 
priate is indicated by the circled letters . 
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The initial step is the Concept Study and conceptualization of the 
course (see Annex C). After this has been completed to as great an ex- 
tent as possible, there is a simultaneous developnent of the course and 
the Evaluation System (see Annexes U and E) and a little later the pre* 
paration for the trial run of the course (see Annex F), It is iji5)ortant 
to eni(^asize that the course and the Evaluation System development should 
occur simultaneously and in close interaction so that i^en certain content 
or implementation methodologieii are developed, the people developing the 
evaluation system will have the task of devising methods of evaluating 
the usefulness and effectiveness of the aspect developed. This inter- 
action between course and evaluation development will produce a more 
precise statement of instruction and course objectives since the developers 
may propose an objective, content, or method, vdiich is too vague to be 
evaluated and thus no information on its place in the course can be 
gathered. 

The progress made in the course and evaluation development provide 
input into the Coach Training Course development. This relationship, 
while not completely diagranmed is very important, [fowever, there ;fill 
always be a lag between the Course development and the Coach Training 
Course development until the Project Formalization stage is reached. 
Once the Course r^d Evalxoation Systems are developed and their inputs 
into the preparation for the trial of the course are made, the next 
step is to try the courses and implement the evaluation system incor- 
porating both fonnative and sumnative evaluations (see Annex F) . 
Since evalxjation is a continuous process, there is constant redevelopment 
occurring iii both the course and the evaluation system. This again is 
an undiagrammed process but occurs naturally since both the course and 
the evaluation system are being tried and evaluated for their redevelopment 
implications. The cycle of conceptualization, development, trial, and 
evaluation is in constant flux. Thus, in the total process of project 
development one simply stops this cycle and formalizes the course or 
program at an arbitrary stage of development. 
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IV. PROCEDURES FOR ffiVELOPING EVALUAriQN SYSTBIS 



This section of the paper provides a more detailed description of 
the steps and stages involved in developing an evaluation system and 
indicates the types of infomation, sources of infomation and the dec- 
ision points. An initial attempt at a behavioural specification of 
evaluation system development is contained at Annex G. In order to 
discuss these procedures more meaningfully, the previous section of this 
paper provided a brief description of the context of the "procedures 
for developing evaluation systems". 



A. Evaluation Model 



Many writers have articulated course development models similar to 
the follov^ing: 



Concept 
Study 




Develop 
Course 




> 


> 



Try And 
Evaluate 



1 



Re-Cycle 



This type of model fails to recognize that development of an 
evaluation system is difficult and time consutiing. Failure to attend 
to the eVi?luation development too often means that the questions on how 
or ^at to evaluate are being asked half way through the trial of a 
course. Indeed, all too often evaluators attempt to obtain evaluative 
information from whatever data may have been accumulated, after a pilot 
course has been completed. 



In addition, this model does not make explicit what is meant by 
'He-Cycle". Certainly they cannot mean that all aspects of conceptual- 
ization, develqanent and testing must be redone! Presunably, the infor- 
mation bbtained will show the effectiveness and efficiency with 
the course has been developed axixi tried and the specific conq>onents vdiidi 
require further development. 



ERLC 



2S9 



model: 



The approach to evaluation for this paper is shown in the following 




Develop 
Course 



Develop Evaluation 
System 



Try And 


— > 


Advanced 


Evaluate 




Development 



This model recognizes the need to develop, concurrently with course 
development, an evaluation system \Aich is directed to obtaining the 
infoimation required to do advanced development. The same approach can 
be used for more advanced developmental stages, i.e.: 



Advanced Course 




Development 






WW 




Develop Advanced 






Evaluation System 





Program E]q)erimentati(ai, 




r 

Program 


Try and Evaluate 


— > 


Fotmalization 



B. Evaluation Requirements 

1. Overview of Total Requirements 



The three primary purposes for developing an evaluation 
system are: 

a. To obtain infoimation for course developers so 
they can make ^ropriate decisions to accept, 
modify or reject developed components and 
concepts (foimative evaluation). 

b. To obtain infoimation \Aiich is required for 
dissemination activities (sitimative evaluation) . 



ERLC 



260 



Cc To obtain infoimation which will aid the users in 
the installation of developed and tested course 
packages • 

It is probably impossible to articulate all the possible evaluation 
infoimation requirements >diich could or should be obtained. Ihe following 
list is neither e^austive nor exclusive of all infoimation required: 

1 . How effective or efficient is the master or 
prime instructional strategy (e.g., IPI» 
groi^ process)? 

2. How closely does the developed course reflect 
the concept e}q[)ressed in the outline design 
specification? 

3. How closely does the developed course fit the 
parameters and bounds descnbed in the outline 
design specification? 

4. How valid and complete is the course content? 

$• How closely do the instructional objectives 
meet the stated criteria? 

6. How realistic are the stated student entry 
d^racteristics? 

7. Lesson Plans/Curriculim Guide and Associated 
Methods and Materials: 

(a) How closely do they meet the stated 
criteria? 

(b) How closely do they result in student 
achievement of the instructional 
objectives? 

Cc) How effective and efficient is the 
student/instructor organization 
(e.g.» specialist^ coaches » one 
coach per groip)? 

(d) What additional or alternative 
methods or materials exist for 
adiieving the instructional 
objectives? 

(e) How effective and efficient is 
the student/coach ratio? 
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8, What is the optimal sequence within each course 
and between associated courses? 

9, What effect does gvoap cai5X)sition have on 
outcomes? 

10. To vAiat extent do the students achieve the 
instnictional and course objectives? 

11. To vAiat extent do the students retain and 
apply the skills, knowledge and attitudes 
learned? 

12. Hew ca]q)atible are the instructional staff 
behaviours with the course objectives? 

13. How appropriate are the standardized tests 
used for the student population? 

14. Ftow well do the adninistrative and sijq)port 
systems meet the needs of the potential 
users of the course package? 



2. Information Requirements Related to Developmental Stages 



The course strategy should be evaluated quickly to ensure that 
It IS effective and efficient before making a large expenditure of time 
and money. On the other hand detexmining the optimal course graming 
implies e3q)erimentation and it would be inpractical to attapt to obtain 
this information from the Pilot Study. TAle 3 is an attempt to relate 
the evaluation infoimation requirements to the developmental stages. 



C. Development Procedures 



(1) (Xitline Design Specifications For Evaluation Development 



The outline design specification for evaluation develc^inent 
is a guide for direction and emphasis to be placed on evaluation for the 
particular project. It should contain: (a) All the infoimation contained 
in the course development design specification; (b) Decision areas to be 
addressed in the evaluation system OR priorities for component evaluation 
development; (c) A direction for the allotment of specific staff mend>erS| 
assistance from other divisions, usf^ of consultants , etc. 
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RELATIONSHP BETVEEN EVALUATION INPOPMATION 
REQUIRBIEI^ AND DEVELOPMENT STAGES 
TABLE 3 



DEVELOPMENTAL STAGES 



EVALUATION REQUIREMEOTS 


Exploratory Development 


Prototype Development 


Pilot Study 


■^ced Development 


[Program Foimalization 


Field Test 


operation. Systems 
Development 


Dissemination 


Installation 


1. Instructional strategy 


w 


a 


X 








> 


c b 


a 


2. Developed course » reflect the concept 


(X) 


a 


X 


a > 


c a 






b 




3. Developed course, fit the parameters 






X 


a > 


c a 






b 




4. Course content valid and conq[)lete 


(X) 


a 


X 


a > 


c a 






I b 


a 


5. Instructional objectives meet criteria 




Cx) 
















6. Student entry characteristics 


(X) 


a 


X 


a D 






t 


I b 


a 


7. Lesson Plans/Curriculim Guide, etc. 
a. Meet criteria 


(X) 


a 

Cx) 


X 


a 

[X) 3 


c a 










b. Student achievement 






X 


a 3 


c a 






b 




c. Student/instructor organization 






X 


a ^ 


c a 




t 


I b 


a 


d. Additional or alternative methods 






X 


a J 


c a 




t 


1 b 


a 


e. Student /coach ratio 






X 


a : 


c a 




t 


I b 


a 


8. a. Seouence within course 










c a 


X 


b 




i b 


a 


b. Sequence between associated courses 








: 


c a 


X 


a 


t 


1 b 


a 


9. Groiq) con|)osition 








: 


c a 






1 b 


a 


10. a. Students adiieve instructional 
objectives 






X 


b : 


K b 


X 




c a 




b. Students achieve course objectives 






X 


b : 


< b 


X 




c a 




11. Retain and apply skills, knowledge 






X 


b : 


c b 


X 




c a 




12. Instructional staff behaviours 






X 


b : 


< a 


X 




c b 


a 


13. Standardized tests 






X 


b : 


ic a 






c b 


a 


14. Administrative and siq)por;t systems 












X 


a 


3 


c b 


a 



OOCE: Stage lAere evaluation information 6b trained 

(x) - logical analysis, professional judgments, opinion, etc. 

X - empirical trials mi measurements of effects on students 

a - stage where evaluation information of primary use 

b - stage where evaluation information of secondary use 
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The specification for a Pilot Course evaluation systan 
might, for example, direct that special emphasis be placed on student 
achievement and on the effectiveness of the developed lesson plans. 
Advanced development at a later date might be directed to obtaining in- 
foimation on the optimal sequencing and group composition, etc. 



(2) Design Specification Study and Conference 



Experience has shewn it to be essential that development 
work should not commence until there is assurance that a consensus of the 
concept and specification is held by both management and developers. This 
includes: (a) A thorough study of the design specification to ensure that 
all assigned staff members have a good understanding of the requirement. 
Clarification must be sought if there are doubts about the specification 
of ideas for constructive improvement; (b) A conference with the management 
to consider any proposed amendments to the specification, to make initial 
decisions on the requiranents for additional staff or for consultant 
requironents, and to ensure through discussion, that the developnent and 
management staff have a clear understanding of the evaluation requirements 
to be addressed. 

The above conference may be held in conjunction with the 
course development design conference. If not, it is important to ensure 
that decisions reached at each conference are passed to the associated 
development staff manbers. 



(3) Input From Course Development 



As strategies, methods, and materials are developed as a 
product of the course development activities, the con?)onents should be 
passed to the evaluation development staff. The diagrams as Annexes D 
and E to this p^r should be read in conjunction with this part of the 
paper. The diagram is not intended to display a process vHiere one step 
must be completed before the next step may be started. In fact, many 
steps can be carried out simultaneously and others may be cyclic in 
nature. The sizes of the diagrams are not related to the time frame nor 
do they give any indication of the amount of work involved in each step. 
The diamonds show decision points vrtiere the development process can be 
given critical examination. Hie diagram is not intended to show a "one 
way" passage of information from the course developers to evaluation 
developers. If the developmental staff members are to adiieve unity of 
effort, there must be frequent conferences and consultation between course 
and evaluation developers, e.g., in developing behavioural objectives. 
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(4) Input From Instruction 

» 

If it is necessary or desirable to develop and conduct a 
coadi training course , the activities should be similar to those described 
in course development and development of the evaluation systm (Annexes 
D and E) with an evaluation system specifically developed to evaluate the 
instructional staff course (coadi training course in Ann^ B) . 

In any event ,it is extremely important that the Evaluation 
System include means of measuring tiie con^tibility of the instructional 
staff behaviours and attitudes to the course requirements. 



(5) Input From Adninistrative and Sigport Systems 

The corporation (Organizational Context) will probably 
develop adndnistrative and stqpport systems sudi as student career and 
personal counselling and infoimation systems. Some unique sii^qport 
systems may have to be developed for some courses. Ihese m^ have 
considerable effects on student achievement and it is therefore in^rtant 
that the impact of these associated or unique systems be evaluated 
(Figure 3, areas 3, 4 and 6). It is also important to realize that vdien 
projects are installed in other agencies , we must be able to identify ndiidi 
of these systems are essential for adoption and which are only coiporation 
requirements. 



0. Evaluation Conq[)onent Development 



(1) Overview 



The diagram at Annex D shows evaluation component 
development as a single activity. In fact^ conqponent development should 
be treated as a series of discrete md ordered activities vdiidi are 
developed in a systematic manner. 

The diagram at Annex E siiggests the following series 
of ordered activities for conqponent development: Oeteimine Decision 
Areas, Deteimine Infoimation Requirements, Detennine Methodology, and 
Prepare Detailed Procedures . 

(2) Deteimine Decision Areas 

If infoimation is to be obtained for decision making, this 
step should include the deteimination of the decisions whidi are to be 
made. On the ottier hand, if decisions have already been made this step 
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should include the recording of these decisions as a basis for si^sequent 
activities. 

Who are the decision makers? For certain major decisions 
this might be the executive head of the organization, who will have 
exercised the perrogative in this danain hy stating the requirements or 
decisions in the Outline Design Specification. For course redevelopnent, 
the course developers should probably be considered the decision makers. 
In decisions made or required in the Dissemination and Installation 
Phases, course development managers might well be the prime decision 
makers. 



The decision makers, not the evaluators, should primarily 
detexmine the nature of the danain to be exanined. The evaluator can 
and should, however, point out inconsistencies, potential difficulties, 
or additional data that might modify the decision maker's views on the 
relevance of certain concepts. 



(3) Deteimine Infoimation Requirements 



The following steps are suggested within this activity: 

(a) Deteimine vdiat infoimation is required . Based 
upon the decision requirements specified in the previous activity the 
evaluator will have to decide vtfiat information will enable the decision 
makers to arrive at apprq>riate decisions and assist in the inrplementation 
of these decisions. 

(b) Deteimine the information is required. 



(4) Determine Methodology 



Ihis activity subsmes a niiid)er of steps: 

(a) Ascertain yihat data must be obtained . 

(b) Deteimine the potential source of the required 
data . 

(c) Design or select the data collection instninents. 

(d) Ascertain idien the data should be obtained . 

(e) Deteimine how the data will be processed to obtain 
the required infoimation . 

(f) Develop suitable formats for reporting the infor- 
mation and the circulation of tne reports. 
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(5) Prepare Detailed Procedures 

To ensure that the required information is obtained it 
is essential that detailed procedures and responsibilities be articulated. 
These procedures should make explicit the timings and responsibilities 
for data collection, the processing required, and the dates or timings 
for submission of data collection instrunents and evaluation information. 



(6) Obtain Approval of Component Development 

As evaluation conqponents are developed they should be 
exanined by the course development staff. When staff consensus has been 
reached, they should be integrated into the total project evaluation 
system. 

E. System Development 



(1) Overview 

It was stated that, "component development should be 
treated as a series of discrete and ordered activities which are dev- 
eloped in a systematic manner". However, as the evaluation components 
are developed, they must be accumulated and integrated. 

The integrated evaluation system must include all 
evaluation components which were developed using inputs from course 
development, from coach course development, from administration and 
support system, and from other inputs (see Figure 3) , 

(2) Evaluation System Package 

At this point in time, no attempt has been made to 
prescribe a standard form or format for the evaluation system packages. 
It should, however, contain an overview or rationale vdiidi describes 
the developed evaluation systems, a detailed set of procedures, timings, 
aiid responsibilities for data gathering and processing, a copy or 
reference to the instrunents which are to be used and details of the 
reports (including timings, responsibilities, format and circulation) 
^ch will be issued to inform staff members of progress. 
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(3) ^roval of Evaluation System Package 



Ihe evaluation system package should be examined by the 
course developnent staff and by the manager. When developnent/evaluation 
consensus has been achieved^ the package should be foxwarded to the exec* 
utive head for further discussion and/or approval. 



F. Conclusion 



Ihe stages of project development and the related systems are 
in various levels of development. Seme systems ("Concept Study", "Course 
Development" and "Course Trial") have been developed in considerable 
detail but most are in a fairly primitive and untested state, especially 
the later stages of the program development. Ihus, several systems have 
yet to be developed \iiidi would operationalize the various stages of 
project development hypothesized. 

Finally, the procedure for the development of evaluation systems 
attempts to make explicit the task areas. However, the explicit relation 
to course/program (tevelppnent and operation are still quite vague and 
unintegrated. Development of an evaluation system to encGnQ>a3S the total 
scope of evaluation indicated in this paper has just been conceived of - not 
by any stretch of the imagination conceptualized. Steele's (1973) analysis 
of the ccmplexity of evaluation, and her atta(iq)t to provide an initial tax* 
onomy of evaluation systems may point to the i^^ssibility of developing 
an all-enconqpassing ultimate evaluation system. At any rate, the science/ 
art of program evaluation is at a more sq)histicated level than ^riien we 
made our first attempts between 1969*1971, described in Part II. 
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V. SPECIAL EVALU/VTION OONSIEERATIONS IN THE DEVELOI*err AND 
CONTINUED OPERATION OF OPEN, BROADCAST, MULTI-lCDIATED, 
INDIVIDUALIZED, CCNTINUOUS PROGRESS, RESPONSIVE 
INSHttlCTIONAL SYSTDB FOR EDUCATING DISADVANTAGED 
AND/OR ISOLATED ADULTS IN BASIC EDUCATION AND LIFE SKILLS 



A« Introduction 



This title is long and involved because I wish to i]iq[>ress on the 
reader that the problem is likewise. In the title **the mediun is (part 
of) the message" since its iqppearance (format) relays the message in a 
visual manner independent of the meaning of the words* The structure 
of this section of the ^paper will be to *\inpack** the title and indicate 
the evaluation problems produced by eadi facet of the title. At this 
time only the areas of problems and the sources for infoimation can be 
indicated. Total description of the specifications for an evaivation 
system would require a lengthy treatment. 

There are several systems in existence vdiich address certain aspects 
of the title. The major ones examined as background for this section 
are the State University of Nebraska (SUN) (see Cavert, 1972a, 197Zb; 
"Procedures for Instructional Design", 1972; State University of Nebraska, 
1973). Rural Family DeyelopP^t (RFD) (see 'The RFD Project", 1972; 
Staith and ingroville, 1972; Udvari, 1972) ; The Open Ifaiversity (OU) o f 
the IMted Kingdom (see Bates, 1972a, 1972b, i9^a, 1973b; Dirr, 1971; 
Lewis, 1971a, 1971b, 1971c, 1972; Mcintosh, 1972a, 1972b; Mcintosh and 
Bates, 1972; Neil, 1970; Komiszanski, 1973) and the Children's 
Television Workshop (CIW; "Sesame Street" and "The Electric Company) 
(see Ball, 197Q; Hemott and Liebert, 1972; Palmer, 1972; Reeves, 1970). 
Each of these projects can contribute to aspects of the total program. 

SUN has special strengths in integrating media aspects into instruction 
and }fAien combined with the SasVitdiewan NewStart course development pro- 
cedures, strong well designed units and modules of instruction could be 
produced. 

RFD sought to deliver combined and integrated individualized Life 
Skills/Basic Education instruction to disadvantaged and isolated people 
with the responsive elements of an "Action line" telephone and home 
visitors. 

nu has developed the most elaborate systems and data processing 
techniques to handle large minbers of widely dispursed students. It 
also incorporates responsive elements of regional resource centers with 
tutors and a set of simner sessions. In additior, OU is specifically 
designed to be a self*correcting continuous development system using 
feedback from the population. Finally, they have considerable practical 
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evidence in the organizational* interaction problems involved in developing 
inulti -mediated course materials (problems in cooperation of content spec- 
ialists among themselves and with media specialists) . 

Crw has developed the most sajMsticated methods of developing and 
evaluating the impact of media on the target population (disadvantaged 
children) . 



B. Development 



The teiTO "development" refers to the process described in the 12 
stiiges of development of new training methods (see Annexes A and B) • In 
addition^ other developmental processes are indicated in diagramatic 
foim in Annexes H and I (the SUN development scheme) , Annex J (a system 
for developing the media part of an instructional unit, frcm Brodt, 1973), 
and Annex K (the system for the Open University, described in Leds. 1971a. 
1971b, and 1971c). 

Inasmuch as detailed instructions on program development alreacty 
exist there is no need at this time to repeat or sunmarize this material. 
However, there are a few general guidelines vihidi should be stated here 
in regard to development/evaluation of media. (Other development/evaluation 
considerations are discussed in the sections dealing with Multi -media and 
Broadcasting.) Since Audio-Visual media are ^(pensive to develop and 
produce, and since they either should be done with high quality or not at 
all, a great deal of thought should go into the planning of their use 
before conmitting much time and experise to actual production. This 
evaluative planning basically involves a clear statement of objectives 
and a clear analysis of the role the various media can play in their 
attainment (see Step 9 of the SUN course design model. Annex H). Closely 
related to this is the empirical evaluation o£ the inq^ct or effectiveness 
of short and simple media presentations (Reeves, 1970). That is, do not 
design a complete mediated course, for instance, and then try it out on 
the representatives of the target population. The consequences of failure 
or inadequacy are too costly. This constraint, of course, does not apply 
to material already available. A suggested strategy, in fact, would be 
to try various available mediated instructional materials on a carefully 
selected and representative sample of the target population to assess their 
appropriateness as a medium for this population. If a sufficient variety 
of media are tried (i.e., sample the materials available) on a sufficient 
variety of the population the results can be fed into an "infbimation 
bank" on the appropriateness of different media for different objectives 
with different people (see Annex H, steps 12, 15 and 17 where infoimation 
on the effectiveness of media goes into "data bank" for general develop- 
ment guidelines) . Other infbimation sources are available in the pro- 
fessional AV lore aiiu literature but it probably does not directly relate 
to the population of interest (e.g., adults yAio are disadvantaged, poor, 
isolated, unsophisticated in media experience, low in education level, 
etc.). The results of R.F.D. field testing on disadvantaged adults of 
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Pilot Programs are given on pp. 120-150 ( The RFD Project) . The results 
of C.T.W. testing are in Reeves, 1970. The attitudinal resiilts of the 
field testing done by SUN are reported in State IMiv. o£ Nebiaska, 1973, 
pp. 248-251. Also on pp» 176-184 there are over 133 guidelines listed 
for the use of TV, axidio and non-broadcast media. On pages 1S9"171 
there is a detailed discussion on the relation of module components to 
levels of objectives. In the OU literature. Bate? lists his j^idgements 
on the instructional functions of TV (Bates and Moss, 1973, Appendix 1). 
Also, Gerlach and Ely, 1971 discuss the various media and their relations 
to teaching and instructional objectives (see especially pp. 282 ff) and 
Topper (1973) provides a procedure for evaluating the appropriateness of 
already developed materials. 

All this infoimation needs to be gathered and stored for use in 
developing f«iture mediated units. Thoa, after a carefully thought out 
rationale is done, a small and inexpensive (relatively speaking) AV 
Module is developed and tested. The results put into the "infoimation 
bank" and related to other lelevant information. 

SUN suggests assessing mediated modules for the following attributes: 
audience appeal, ccnqprehensibility, learning, manorability and suggestions 
for improvement (State University of Nebraska, 1973, pp. 228-232). 

In sumiiary, the major point with regard to this development is the 
5 ;rious use of systenatically accumulated knowledge and the intact of 
mr.dia on the target population. Available material should be selected 
to find out if, in fact, it is appropriate or effective for the target 
population and then develop short media modules for use in an audience 
reaction study. If the study is successful then a more extensive field 
study can be undertaken on this same module using circunstances v/hidi 
approach actual conditions in the fully developed course (learning 
envirorniisnt in the field). 

In outline the se^ueoco involves obtaining audience reactions for: 
mediated instructional modules of 1 to 15 minutes in length, mediated 
lessons v*ich are self-contained of 30 to 60 minutes in length and 
mediated course of 30 or so hours. 

SUN field testing procedui^es and results are discussed in their 
report, especially on pages 53-53 and 228-239 and need not be repeated 
here. 



C. Continued Operation 



The major so urce for this section is the Open IMiversity material 
since the OU vi^j explicitly designed to be self -corrective and self- 
improving. To this end, elabi rate feedback and data processing systems 
have been developed. The requirements of the feeAack system are 
sket^^ed in Bates, 1972a, pp. 8-9, the foms used to gather the data 
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from students and field tutors are provided in the appendices. The feed- 
back system using the iitors at the regional resource centers is discussed 
on pages 3-4, A more complete feedback system using group discussions, 
"Course Unit Report Foims", telephone interviews and students' letters 
is described in Bates, 1973a, pp. 8-10, and 15-17. The problems of the 
logistics of producing and distributing the materials is discussed in 
Dirr, 1971, p. 17 and Bates, 1973a, pp. 6-8. 

The feedback system must provide information on both the course/program 
materials and the actual operation of the organization in development, 
production and delivery of programs. This involves the inter-phase of 
area 4 with areas 5 and 1 as they affect students (Figure 3) . 



D. Open 



"Openness" implies that tiie clients will, be varied and nunerous^ 
especially if the program is a successful one. For instance, the "Open 
IMversity" of the United Kingdon has approximately 50,000 students 
enrolled. These students are located in ail areas of the country, v*iich 
in England involves considerably less distance than is possible in Canada. 

The types of people to whom this program will be of interest include 
people desiring part- or full-time study programs, workers who need 
additional training for advancement, families who wish to study together, 
housewives wishing training to re-enter the labor force, and people who 
have been away from a fonnal educational environment for some time and are 
hesitant to rejoin the standard educational system. Tlie Large I :>CT?iiicition 
will also include people receiving income support fran diverse agencies"^ 
such as; Maiq)cwer and Immigration, Unemployment Insurance Comnission, 
Welfare, Indian Affairs, provincial suK>ort programs, etc. 

Considering the diversity of populations involved and the diffi- 
culties in receiving broadcasts in certain locations, no one system is 
Adequate to deliver the necessary information to the studSit. 

f Bates (1^73b, pp. 4-5) has described the procedure of defining the 
necessary rhdjcritical versus optional characteristics of the target pop- 
ulation to vrfi^ the program of the Open University is directed; SUN has 
described their efforts at defining the target population on pp. 144-157 
in their repbrt, and RFD procedures and results are contained on pp. 144- 
170 and 255-264 of their report. The recruitment and promotional pro- 
cedures of RtD are described on pages 41-54. Since the RFD target pop- 
ulation is similar to the one in this section, their information is most 
appropriate to use. 
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The use of broadcast media is a major mechanism for making the 
system open (accessible) , although open refers mainly to the opennesr> of 
the "entry bridging systems" (area 3 in Figure 3) . Using broadcast media 
means that the program potentially has wide exposure in the "social/ 
cultural context" (see Figure 1 and area 1 in Figure 3} . This in turn 
creates problems in defining the population to use wh^ evaluating the 
impact of the program. The "target population" defined in the project 
description and the actual population the project reaches may be quite 
different. This is one of the problems of the Open University where the 
inteni!<?d target population is the large niDd>er of people v4io, while C2q>able 
of uniA'ersity level education, dropped out for a variety of reasons. Their 
actual population (registered students) turns out to be people ^o are not 
particularly educationally disadvantaged but find the Open University a 
convenient way to further their education* The program may attempt to 
reach "disadvantaged" individuals but, as most people have worked 
with the "disadvantaged? ' know, this is a most difficult groi;p to reach. 
They tend to be socially isolated from the 'taain stream" society and 
broader cultural inputs or else be influenced by the less desirable media 
aspects of the larger society. In addition, much of the purpose in using 
mass media is to reach individuals physically isolated, or remote from 
educational opportunities; that is, people \dio can not take advantage of 
educational opportunities because of 0iysical distance or inmbbility. 
However, in some instances, the isolation may also mean isolated from 
access to mass media and electronic comrnmication systems. 



E. Broadcast 



The problems in the use of broadcast media, alluded to in the prior 
section, have been much researched and analyzed by the Open University 
(see Bates, 1972a, 1973a, 1973b; Bates and Moss, 1973; and Mcintosh and 
Bates, 1972). Because of the present and projected difficulties in using 
the BBC, the Open University is reexaminiiig its use of broadcasting for 
instructional purposes (Bates 1972a, 1973b and Bates and Moss, 1973 are 
particularly geimaine) . In Canada, there are few if aiy channels avail- 
able to use for educational purposes at this time. Should they become 
available, then the tise of broadcast media (especially TV) will be more 
feasible. 

While the broadcast media reach many people and are thus good for 
promotional, enrichment » and special events activities, it is another 
matter to use them as an integral or required part of the instruction 
(Bates, 1973a). A comparative stu(fy of various media delivery options 
is contained in Bates, 1973b. The RFD use is described in their report 
.n pp. 55-67 and the Report of External Evaluation Team (beginning after 
p. 282) provides a short evaluative statement on broadcast usage on pages 
1M9. 
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The two major problems involve the availability of broadcast time, 
both miiiber of hours and time of day, and how these times fit with student's 
available time. For instance, if many students work during the day then 
the mid d^ hours are of limited use. These two constraints (available 
network time and student access times) presently limit the usabilxty of 
broadcasting as a required and integral part of instruction (Bates, 1973b, 
pp. 18-25) . There is an additional problem of \Aether the population 
of interest (disadvantaged and/or isolated) has sufficient access to the 
mass media to warrant use of broadcast delivery. The Qud)ec system, '^tilti- 
Media", has decided to use alternate methods of delivery of instruction 
due to these types of problems. 

The Open University research did find that the broadcast aspect of 
their programs produced a pacing function (Bates, 1972a, p. 15) - i.e., 
helping students to pace their studies to keep up with the assignment 
and testing dates. 



F. Multi Mediated 



Most systems using the '^open £?)proach" also use a variety of methods 
of instructional delivery. The aim is to include optional, multiple and 
mutually siqpportive methods of taking a given course. For each mediun of 
instruction, certain instructional taslcs are set out. Parallel courses 
are developed where the instructional tasks can be acccn^lished by more 
than one mediun; in other words, a system whose conqxment parts can be 
put together in different ways depending upon the needs of the populations 
and transmission and reception facilities for broadcasts. Needless to s^, 
this is easier said than done. The intention of the multi media approach 
is to solve the •'problem of match" between the learning styles and pre- 
ferences of students and the manner in \rfiidi the learning materials are 
presented. The assiii5)tion of this approach is that if tlie st dent is 
provided with enough alternatives, at least one will be optimun for his 
particular needs. 

The use ' a variety of media (print and non-print) creates course 
development^ *lems. For instance, subject matter specialists tend to 
be print oriented and unfamiliar, indifferent, or hostile to using other 
media in delivering content. On the other hand, the media specialists are 
unfamiliar with content areas although they knew the process -d^livery-mediun 
characteristics. Thus, using multi media delivery of subjects requires 
considerable cooperative effort among experts who do not ordinarily inter- 
act. 

The Open Universit)'^ has had considerable problems producing an inte- 
grated multi media program (see Bates 1973a, and Bates and Moss, 1973). 
The use of multi media in the RFD program is described on pp. lb- 40 of 
their report. One of the major problems, related to an integrated medj.ated 
instxuctional package, is deciding \Aiat media serves v^t objective. SUN 
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has made detailed suggestions as to hcM each component of their lesson 
module relates to the objectives of the lesson (State University of Neb- 
raska, 1973, pp. 157-174) and Gerlack and Ely have suggested the in- 
structional uses of various media (pp. 282 ff) • As mentioned in the 
development section ("A") of this part of the paper, this area requires 
careful analysis and evaluation. There is a danger that the prograns 
will be multi media just because it is "an in thing*' to be (i.e., "Good 
is better than evil because it^s nicer" - Mammy Yokun). 

An initial inventory of possible media conponents might be as follows: 
Visual imits : film, film cartridges, film strips, video cassetts, broad- 
cast, cable or closed circuit television, slides, still pictures; Audio 
units : radio, audio t^, audio cassette tapes ^ Action units; exercises, 
experiments, problems, simulations, workbooks, self testing; Print units: 
commercial newspaper, bulletins, student newspaper, correspondence syllabi, 
stxxdy guides, achievement examinations, books, texts, readings, homework 
assignments; Responsive units (providing opportunities? for interaction): 
action line (telephone-radio combinations), telephone infomation, tutoring, 
testing, surveyr and assistance, regional learning centres, tutorials in 
local centers > opinions and evaluation from students (feedback), ^tudy club 
groi^, weekend or other classes, short simmer sessions, home visits; 
Miscellaneous units : coiqniter assistance (data processing, scheduling, 
grading, instruction), degrees and certification. 



G. Individualized and Continuous Progress 



The Training Research and Development Station in Prince Albert has 
individualized the basic education course , Learning Individualized for 
Canadians (LINC) , which would be used as the basic curriculun in this 
program and the life skills component is presently being individualized 
as mudi as is feasible using a peer teaching approadi. Most 'open' programs 
require independent study which is one aspect of individualized nnd con- 
tinuous progress programs; however the two are not the same. Ihus, the 
Open Ihiiverrity courses while requiring "independent study", are paced by 
due. dates for tests and assignments, broadcasts, the distribution of 
materials anc course starting dates (registration). The RFD program 
(see pp. ^9*^M) closely approached total individualization since the 
individuals selected units of interest in the "Content Centers" curriculun 
(similar to Life Skills) and from there went into the more academic (basic 
education) aspects of conomjnications and math \dien tiie individual discovered 
he lacked skills needed to cope with the "content center" material. 

One problem discovered in Open University researdi i^ich will be 
more problematic in ^disadvantaged^ populations, is the students' lack 
of basic learning and study skills needed for independent study. This 
increases the importance of the * responsive' aspects of the systm (dis- 
cussed in the next section). Ihus the home study component of tnese 
approaches irfiich is one of their raajor strengths is also a source of 
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problans. More en5)hasis must be put on developing and evaluating support 
services and stitttent feedback components of these "instructing at a distance" 
programs than in 'direct instruction* (face to face) delivery approaches. 

Scheduling will also be a probl«jm in light of the broadcast times 
constraints (discussed in section 'D' above). That is, in \diat ways can 
a mass broadcast of material be coordinated and integrated in a program 
ifAieTe the individual can enter, progress, "step out", reenter, recycle 
skip, etc., in his own individual and unique pattern of progress. 



H. Responsive 

The tern "responsive" refers to those aspects of an instructional 
system which are responsive to student requests and needs (a personalized 
conponent). Most aj^roaches have found this aspect essential for the 
operation of "instmction at a distance" prograns. Thus, the Open 
Itaiversity has elaborate feedback/data processing con^Kxnents. To 
increase the responsive and personalized aspects of the program, they 
established study centers with tutors (discussed in Lewis, 1971b, p. 112; 
Bates, 1972, p. 9), suniier schools (Lewis, 1971b, pp. 112-113; Bates, 
1972a, p. 10), and a student newspaper (Bates, 1972, p. 3). The Open 
University has thus done research into the source and type of student 
difficulties (see Mcintosh, 1972a, pp. 35-37 and 1972b), the work load, 
and the inqwict of programs on the students' lives (Mcintosh, 1972a, 
p. 44). 

The RFD program contains two elements of responsiveness "Action 

ffl l^l^,^j.^5^ ' PP- ^ Visits" (The RFD Project, 

PP- 95-Jd3^ aittd^iA nas proposed a WS Telephone Line (State Un&rersity 
of NebraSklL iBTi.ipp. 179-171). . r ^ 



I. Instructional System 

This, of course, involves all aspects of the syston but th^V 
special problems identified. For instance. The Open Univers^\ty ha5 
identified problems of "instructing at a distance" mentioned v«bove (see 
Lewis, 1971a, pp. 4-5; Mcintosh and Bates, 1372, p. 18P^. In systems 
idiich involve certification, probloas of assessing stucu^nt achievement 
(tests and hcane work assessment, grades and certification) and developing 
conparable weights for these components arise. These are problems in 
any educational system involved in student assessment but special dif- 
ficulties arise \Ai&i designing achievement tests for courses developed 
by teams, with instruction and assessment by a variety of tutors ard test 
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readers -graders (including computer scoring). These are discussed in de- 
tail by Lewis (1971a, pp. 6-7; 1971b, p. 113; 1972) and by Mcintosh and 
Bates (Bates, 1972a, pp. 3-4; Mcintosh and Bates, 1972, pp. 196-197). Many 
assessment problems are solved or avoided if the instructional materials 
(e.g., LINC) have been developed using behavioural objectives ^ere methods 
of assessing adiievement are developed hand in hand with materials to 
attain the objectives. 



J. Education in Basic Education and Life Skills 



The best available model for an **ppen approach" to this type of 
curriculun was developed by RFD in their "content center", basically a 
cognitive ^roadi to life skills, inlying the life skills conc^t as 
descidbed in Readings ui Life Skills in an open mediated system is another 
matter and to do so with the "disadvantaged** conQxninds de pnent/ 
evaluation problans. This again eiqphasizes the very i]q)ortcuit aspect of 
"responsiveness" mentioned in section 'G*; the use of tutors, home visitors, 
action line, study grot4>s, peer teachii^g and so on. 



K. Disadvantaged and/or Isolated Adults 



The last topic in the title is the actual focus of the progran: 
The Target Population. Saskatdia/an NewStart and Hie Training Research 
and Development Station have considerable infoimation on the dempgra^iic 
personality and abilitx aspects qf th6 target population but little 
infoimation on trie use vaflpus medi^ with "disadvantaged" fulults . 
^ RFD has done relevant woik he^^ ( The Rp) Project^ pp. 119-1561 171-174) 
' and SUN aiq^diasizes the carei^^r definition of the target population and 
the selection of individuals from this population to test out materials 
developed. The CIW procedures provide guidelines for this aspect (Palmer, 
1972 and Reeves, 1972) although their population was pre-sdioolers. 

There are special prdblons of program delivery in using media with 
\ the disadvantaged/isolated since part of their "disadvantageness" involves 
'ijbeiiig cut off from available media. Tnus, the problem needs special 
stiK^ to describe the types of delivery conponelliits \itddi are most feasible 
for people in certain settings. For instance, losing advanced. AV technology 
will be a waste of time and money if the pff^aple are unable to operate it 
or use it to its full potential. This means that not only must the target 
population be well defined but the learning situations in vAiidi they op- 
erate must also be specified and the delivery sy<>tem characteristics 
matdied to both the characteristics of individuals and sirjations. 
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L. Contextual Evaluation 



This area of stu^ involves evaluating the effects of a system 
such as is indicated here on the lai^er context (see Figure 3, area 
1, and Figure 1, "An 'Evaluation Space*", Part I.C.S especially the 
descriptions of the social context, the population and clients). The 
introduction of advanced technology and edicational delivery systems 
into a remote and isolated social setting will probably have major 
repercussions viiidi need to be studied. It is difficult to predict 
lAat they will be but there are anthrqpological studies on the effects 
of sudden introduction of advanced technology into a ••primitive" 
society. Also, the project will stimulate and be stimulated by the 
general movement toward a •XJlbbal village" which McLiiian talks about 
as well as attempts to establish public media netwoiics in Canada. At 
present, these are rare but there is a growing pressure for their 
establishment. The wide spread use of systems such as these will have 
an impact on Federal-Provincial relations in the fields of education 
and ccmnunications vdiich should be evaluated. Thus, introducing sudi 
a system can have effectf on eveiy level of life from establishiqg a 
ccmmmcation link with m isolated individual or ccnmiunity in the north 
to changing the federal -provincial si^res of control and influence. 



M. Conclusion 



As stated in the introduction, only the areas of evaluative study 
have been indicated along with specific reference to relevant literature. 
There is considerable literature available on how to develop a modiated 
module of instniction ''see Annex J). The major areas in need of stufy ( 
are these: 



t 



1. Kow does the tax^get population respond to and leatn 

from the material develaped. ^ 

2. Hw can advanced teduiology be adiqpted and adopted 
to tedmolpgically prinitive situations? 

3. What i]iq)act does this have on the wider social/cultural 
context? 

4. How can m '^instruction at a distance" system be made 
responsive and effective lAen the distances covered 
are very large and die climatic conditions are extreme? 

5. What siifiport systems are needed to insure that tie 
programs developed have a reasonable probability of 
achieving their intended objectives? 
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In the extreme situations of isolation and disadvantaged, the support 
aspects of a program will probably have as much impact on the outcomes 
as the actual courses developed. Thus, in teims of the 12 stages of deve 
loproent (Annex A) those stages of 'IField Test", •K)perational Systems 
Development", 'Demonstration Project" and "Installation" are in need of 
more carefiil srudy. Most of these questions deal with issues that are 
in addition to the problems of actually developing a given course of 
instruction. These problems are difficult enough, particularly lAien 
they encompass Life Skills' types of objectives. Hwever, they are 
coi^xnind^d \Aien they are placed in the context of an open broadcast , 
individualized, and continuous progress system with a target population 
of disadvantaged and/or isolated adults. 
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8TAQE8 OF DEVELOPMENT 
OF NEW TRAINING 
METHODS 

CONCEPT STUDY ~ review of the problem area to 
determine the requirements of the situation and to 
identify goals. Theoretical and research literature are 
reviewed to assess various theories and methods of 
intervention which have been attempted. The result 
is preliminary specification skills and other factors 
involved in determining broad strategies to achieve 
goals. 

, , !>HATORY DEVELOPMENT - preparation"^ 
initial program strategies, methods and materials and 
the evaluation of their feasibility and limitations. The 
result will be a more detailed specification and cost 
analysis and may indicate a reformulation of the 
concept study. 



TYPICAL DURATIONS 



YEAR 



DEVELOPMENT — preparation of de- 
tailed program strategies, methods, materials, evalu* 
ation systems; training instructional staTf, estimating 
^sts and scheduling time and resource requirements. 



T STUDY — test of the prototype which allows 
sufficient acquaintanceship with the problem and pro- 
totype to permit necessary reformulations and/or 
specification of logical alternatives. 



ADVANCED DEVELOPMENT — redevelopment or fur- 
ther development of the entire program. 

PROGRAM EXPERIMENTATION ~ formally structured, 
systematic, experimental efforts to test and evaluate 
alternative program elements with different groups and 
under various circumstances. 

PROGRAM FORMALIZATION — preparation of a tor- 
mal model program which can be used elsewhere with 
predictable results. 



FIELD TEST — of program model under brdinary 
operating conditions to determine program support 
services required by the user and the interaction with 
other programs and services in the user institution 
or communir 

OPERATIONAL SYSTEMS DEVELOPMENT — prepare* 
tion of the operational systems and procedures for 
implementers, administrative support personnel and 
the monitoring agency expected to use the new 
methods. 



DEMONSTRATION PROJECT — major attempt \o 
fester adoption of the new program including jcint 
sponsorship of the project by a potential user agency 
ir which the operational systems are used. 



DISSEMINATION — may take place concurrently with 
previous stages and involves publicity, seminars, con- 
ference presentations and publications designed to 
secure full support for widespread adoption by acade- 
mic, professional and administrative reference groups. 



INSTALLATION — this final stage includes provision 
of consulting services and staff training necessary for 
satisfactory program adoption. ^ 



INTENSIVE DEVELOPMIENT 



ONQOINQ DEVELOPMENT 



280 



' U. wnwtuijim 



I > iff J 




lwy..^filwa^jigu^toJtliyi^|il 



HI MW Um—I MCT' 



il 



lug. U IWL. iTWl 



S> S> VS. V 



* i ^ ^ ^ ^ 



A A --C 




CSam to Wf Ltd I 
mr 2*3. lauMfOir * namtn. an. 




u 
M 

ucr 



Un fKlUt OOOMt 

itfic muxim oontt 

un KlUt OOMl IMIimB 



\4tViKnm fttTBf KV. HiTDI 



ERIC 



282 



CONCEPT STUDY 




283 



Course and Evaluation System Developnient 
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ANNEX E 

FVAIIIATIQN COMPONENT DEVELOPMENT 



EEVELOPED 

Materials* 
Strategies, 
Methods, 
etc. 



DETEFMINE 

SECISION 

AREAS 



DETEFMINE INPOBMATION REQUIRBenS 

1. Infoxmatior is required. 

2. When is it needed. 



DETEBMINE METODDOLOGY 

1. What data is required? 

2. . Source of data. 

3. Design or Select Data Collection 
Instnments. 

4. When should data be obtained? 

5. How is data to be processed? 

6. Develop reporting format. 



DEVELC^ 
PROCEDURES S 
RESFONSIBILinES 
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COURSE/PROGRAM TRIAL 



AIMEX F 



CouTM and 
Evmluation Sys- 
teas Prat 
DevelopMit 
Systaas 



Approv)^ 
'Provisic 
Lai Materials 
^ Evaluation 
Procedures 



Initial 
Trial 

Direction 




Produce Draft 
%)ecification 




Distribute 
Approved 
%)ecificatiQn 




Prepare Detail- 
ed Trial Plan 




Conduct 
^Intake 
Process 



Develop 
Udqye 
Proced- 
ures 



Acquire 

'ttiique 

Resour- 



ce 




Prepare 
Final Report 




AOTANCED 
DEVEL!XMDir 



Identify 9 Select 
nstnictional Staff 



Train and Orient 
Instructional Staff 
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EVALUKTION DEVELQPMEWr OBJECTIVE 



Given: (a) an outline design specification for a course to be 
developed (or redeveloped) or a developed ^.corse package, (b) an 
outline design specification for evaluation development, (c) the 
developed course conponents, (d) ai^~ standard adiievenent tests 
to be used, (e) developed siq;iport and adninistrative systems, and 
(f) any other infonoation or materials gennane to the requiroBent. 

Then the evaluation development staff will be able to: 

a. Deteimine and obtain acceptance for aay other decisions 
idiich vdll have to be made. 



b. Deteimine the decision points and timings ndiere infor* 
mation and alteinatives muse be provided to: 



i. 


Managers of the Corporation. 


ii. 


Manager (s) of the particular project. 


iii. 


Course Development Staff. 


iv. 


Evaluation Development Staff. 


v. 


InstTucticnal Staff. 


vi. 


Students. 


vii. 


Staff responsible for suj^rt and adninistrative 




systems. 



c. Deteimine the information will be required to 
make the decisions. 



d. Deteimine the data and processii>g vdiidi is required as 
a basis to arrive at the required information. 

e. Develop or select appropriate data gathering instrunents* 

f. Develop the procedures, timings, responsibilities, etc., 
required to manage the acquisition md processing of 
the data requirements. 

g. Develop the foimat(s) , circulation and timings to ensure 
that the requited infoimation and alternatives are avail* 
able for the predeteimined decision points. 
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h. Prepare an integrated evaluation plan irfiidi will, in 
conjincticm with the developed course prototype , fom 
the basis for a Project Trial Specification i^ich will 
ensble the instructional staff to bbtaini use» and 
circulate the information and alternatives required 
for effective decision making. 

It is assumed that: 

a. The outline design specifications for course developnent 
and for evaluation developnent will contain time or other 
restraints , points of oqphasis and priorities , particular 
decisions idiidi will be required, mid ai^ other criteria 
required to guide the developnent activities. 

b. The course development and evaluation staffs will have 
adequate comnunication and management to ensure unifi- 
cation of effort. 

c. Staff development will be carried out, as necessary, 
to ensure CGnq)etency to achieve the objectives. 
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AMEXH 



PROCEDURES FOR INSTRUCTIONAL DESIQN * (SUN) 



^••ftflMiiit*f**a 



: Individual 



Learning 
ExpecUtlons 



f 



Learned 
Coi^Mtencles 



Instruction 
Required 



Other 
Potential 
Instruction 



Adequate 
Present 
Instruction 



Intent to 
Mediate 

Instruction 



Set 

fioal 
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aem iwivERsm cdurse develofwkt (from lwis, id modified 



START OF 
PHASE 1A 



collate and tnilyse 
Inform tlon about 
coursts needtd. 

£1 



RtvlM Staff talent 
and Inclinations. 

Otan 



Ravi ON budgetary 
constraints: 

- Dean or CTC 
■an/diQfs par unit 

- CUA Mn/d^/un1t 

CT 

- lET Mn/days/unlt 

CT 

- BBC Mn/diiys/un1t 

CT 

- Htdia dtvelopiwnt 

cost/unit 

CT 

- Hadia production 
cost/unit 

CT 

- Picture search 
cost/unit 

CT 

- Library cost/unit 

- Copyright cost/ 
unit 



RtviM books 




available 


CT 





collate and analyse 
Infofwatlon about 
learner character- 



Select broad area 
and tentative title 
of course 



Vrfie 2 - 3 

of course as 
It for prospectus. 



^ suMiary 



Revieif prerequisite I 
IcnoMledge require- ^ 
bants. r 
\ SuhCoB/DaanI 



nsr^sTof— 

rad. 

SubCow/Daan 



:onsuU otiMr 
racultlas on Inttr- 
raculty coursas. 

Daan 



Ihaaa 1 



FlMM 2 
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Plming 

lA - Gourse Plnniiif 
11 • Uiit Plming 
Writins 

2A - thit lAritlng 
21 • DmloiMntel Tasting 
2C - Extemal Asmsaent 
Editing md Printing 



m Audiance Raection Pml 

CT Course Taa« 

CIC Course Taoa Chilmn 

CUA Course Uiit Author 

m Draft 1 of Unit 

K Draft 2 of thit 

D8 Draft 3 of Uiit 

Dt Draft 4 of Ihit 

BA Exumal Assessor(s) 
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COURSE PUUWIN6 



Design conceptual 
■odel of course 

CT 



Review of llsi and 
2-3 page smmry 
Dean 



Compile list of 
blocks and units 
CTC 



Assign blocks to 
working groups and 
units to authors or 
groups of authors 

a 



Decide whether 
consultants will be 
eii|>1c!yed to write 
some Material 

CL. 



Ed Editor 

TET Inst. Ed. Technology 

Maw Msdie Evaluator 

FO Pii>lishLig Officer 

SC Student Consultants 

Sub/con Sub-GoMittee 

NG Norkinf Group 

VC Vice-duncellor 
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mm K 

(cont'd.) 



END OF 
PHASE 1A 



Draw up Noming 
Group schedules 
ML 



Drew up author 
schedules 



Draw up tenutlve 
production network 
end schedules 
OL 



Consult Project 
Control Officer 
CT 



Send tentative 
production network 
and schedule to 
VC's office 
Media Division 
BBC TV and Radio 
lET 

Data Processing 

Dean 



Agree tentative 
production network 
and schedule 
SL 



Review units fro* 
other CTs 

CT 
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JJjgJ % UNIT PLANNING 



List "set books- 
for publishers 
Dean/PO/CUA 



List "recommended 
books" for 
publishers and 
libraries 



Select consultants: 

- agree fees 

- agree work 

- specify source of 
objectives 

- specify 
constraints 

Dean/CT 



List for each unit: 

• objectives or 
contents of 
correspondence 
unit 

- Kty questions 
that student 
should be able to 
answer after unit 

- Items for which 
kits are essent- 
ial/desirable 

- Items for which 
radio Is desir- 
able/essential 

- Items for which 
summer school Is 
essential 

CUA 



Prepare tentative 
conceptual model 

iaii 



specify special 
editions 

* size of run 

- probable 
publisher 

- contractual 
arrangements 



Prepare first 
correspondence 

Prepare first ra 
outline 



Prepare first TV 
outline 



CUA 



Specify reader 
material (If any) 

- size of run 

* cost of printing 

- alternative 
sources 

* cost of copyright 

- format 

* editorial 
responsibilities 

- linking materials 
*- contractual 

arrangements 
CT/PO 



Clear copyright on 
reader material 
(If any) 

QL 



Specify supplement- 
ary materials 
(green, blue, pink 
or purple pages) 



Phase IB 



Produce 1st 
oufline and 
tentative 
nocbl of 
corrospondence 
m%^erial 



1 



2 




pxx>duce 1st 
outline of 
T.V. 

progranne 
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ANNEX K 
Ccont'd.) 



END OF 
PHASE IB 



Amend outlines, 
conceptual model 
anci selected lists 



Comment on material 
from author 

UG 



Submit outlines, 
conceptual model 
and selected lists 
to Working Group 

CUA 



Amend material 



CUA 



Connent on material 
CT 



Subjfilt amended 
material from UG 
to Course Team 



CUA 



Submit amended 
material from WG 
to student 
consultants 



CUA 



Comnent on material 
SC 



Amend material 



CUA 



Revise 2*3 page 
summary of course 
as If for prospectus 
Dean 



Review prerequisite 
knowledge requlre- 
nents again 

Oean/CT 
■^Jtevlew budgetary 
Constraints (again) 
Oetn/CT 
Review extent of 
integration required 
(again) Dean/CT 



Identify and engage 
suitable external 
issessors (EA): 

- specify objectives 
of assessment by 
EA 

- specify contractua' 
arrangements 

- send tentative 
schedule of 
deliveries to 
units 

Deal 
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START OF 
PHASE 2A 



IMT WRinNG 



Sj^ify course team 
rules for formt 
and layout 

CTC 



S|)ecify house rules ! 
tor typescript draf tl 
PO I 



Review, extend and 
mmodp -for each 
uiit: 

- objectives or con- 
tents of corres- 
pondence init 

- conceptual nodel 

- key questions 

CUA 



Review » extend and 
anend» for eadi 
init list of items 

- to be tau^t in 
smmr schools 

- requiring kits 

CUA 



Specify facilities 
available in study 
centres 





Specify activities 
of face -to- face 
tutors 

CT 



Review, tTopmd and 
amd 

' television outline 
- radio ojtline 

CUA/BBC 



Write first full 
draft (M) of cor- 
respondence 



Select TV 
presenters 



Prepare^an assess- 



■ent eqihasis 
matrix 



GUA 



List material 




needing copyri^t 




clearance for de- 




velopmental test- 




ing 




CUA 





Pre 
moc 


pare nxqM or 

KUQS of ffTflnhiri 




required for cor- 
respondence text 
OIAMIEV 





Select radio 
presenters 



Oesi^i and develop 
demonstrations for 

TV au 








Design, etc. 
Radio 





Phase 2A 



"I 



produce 1st 
^ill draft of 
jorrespondence 
■aterial 



review ( 
expand 
proposals 
for T.V. 
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ANNEXK 
(cont'd.) 



END OF 
PHASE 2A 



List contents of 
stu4y guide 
Special WG 



Prepare Dl of 
study guide 

Special UG 



Conment on Dl or 
study guide 



CT 



E 



Amend Dl 






CUA 



Conment on Dl 



UG 



It of to 



Submit 
Working Group 



CUA 



I 



Amend Dl (to 
produce D2) 



CUA 



Conment on Dl 



CT 



I 



Submit amended Dt 
to Course Team 

CUA 



T 



Construct comput- 
er-marked quest- 
ions/exercises 

CUA/IET 




Construct self- 
assessment quest- 
Ions exercises 

CUA/IET 


Construct tutor- 
marked questions/ 
exercises 

CUA/IET 





Design and develop 
experimental kits 
CUA 



Select records/ 
tapes for students 
CUA 



Secure copyright 
clearance for 
developmental 
testing 

CT/PO 



GRAPHICS 



GRAPHICS 



Obtain details or 
possible collabora- 
tors for develop- 
mental testing 
sample from 
Admissions Office 
or elsewhere 

lET 



Draft, type, 
reproduce and mall 
Initial recruiting 
letter to possible 
collaborators & 
ARP 

ISL 



TV 



Monitor replies 
and compile list 
of collaborators 
& ARP 

lET 



RADIO 



ERLC 



303 



START OF 
PHASE 2B 



OEVELOFICNTAL TESTING 



Anend 01 of 

study guide 
special WG 



Type* reproduce and 
collate 02 course 
units for develop* 
mental testing 
* CT 




Collate, package 
and mall all mater- 
ials for collabor- 
ators 

lEL 



Trial written 
material 



RECORDS/TAPES 



KITS 



Compile feedback 
questionnaires on 
attitudes, time 
taken, etc. 

lET 



Select ARP 



1 



1 



Test TV ARP 



To redesign If 
needed 



Test Radio ARP 

I 



niase 2B 



produce drift 
2, test, and 
agree 
ravisions 



1 



produce 

detailed 

script 
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ANNEX K 
(cont'd.) 



Assess ansi^ers to 
open-ended questions 
asked during 
developmental 
testing 

CUA 



Arrange for special 
rewards (If any) 
for collaborators 

£1 



ERLC 



Collate* analyse and 
report to course 
teans on feedback 
from collaborators 
In co-operation with 
course unit authors 
lET 




Amend course unit 
(and anclllarles) In 
light of develop- 
mental testing 
analyses (to 
produce 03 
• CUA 






« 


Send 03 to editors 
Dean 





RECOtmS/TAPES 



KITS 



Draft » type, 
reproduce and mall 
thank you letter 
Including note 
prepared by the 
Registry of guaran 
teed admission or 
fee remission 
gained* If ar\y 

lET 



Inform Admissions 
Office of guaran- 
teed admissions 
and fee remissions 
gained 

lET 




Consider need for 
TV and radio * 
commentaries 



Phase 2C 



* 


7 


► 


produce draft 
3, and (if 
necessaxy) 
draft 4. 
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START OF 
PHASE 2C 



EXTERNAL ASSESSCNT 



END OF 
PHASE 2C 



rrepare or connis* 
slon preparation of 
supplementary Mater- 
ials (coloured pages) 

tmui 



Select coloured 
to editors 



Amend course unit 
(and anc111ar1es)1n 
light of EA coRwnt 
(to produce D4) 

. aium 



Coonent on course 
unit (plus ancillary 
1e$) 



Send course unit 
Cplus anclllarles) to 
estemal assessor 
Dean 



Discuss editorial 
policies 

sm 



Approve forMlly 04 



Clear copyright on 
all Itens In 04 



PC 



Specify all graphics 
required for 
correspondence text 
C'JA 



Send graphics specs to 




Media Development 




Dean 






Prepare detailed 
radio script 

mm 



Prepare detailed TV 
script 
^ CUA/BBC 



Collect radio 
material 



Shoot or relect TV 
inaterlal 

CUA/BBC 



Rehearse radio 
progranme 



Kehearse TV 
programne 



CUA BBC 



ERLC 



306 



AWECK 
Cccmt'dO 



START OF 
PHASE 3 



EDITING AND PRINTING 




Construct final 
issessnent (exMln«« 
tlon) questions/ 
exercises 

IM 



Prepare notes, for 
each unit, for 
regional/face- to- 
face tutors 
OIL 



Prepare glossaries 
or handbooks for 
students 



Manuscript edit D4 



Ed 



REOOiqB/TAPES 



KITS 



Send graphic^ 
(conpleted) to 
editors 



HOev 



Produce radio 
prograime 

Dean/CUiUBBC 



Audit radio playback 
CT" 



Produce TV 
progranMe 



View TV playback 



CT 
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EDITING AND PRIKTING 



Stnd final assess* 
■ant to external 
exanlners 

Dean 



Send final assess^- 
pent to Nedia 
Production 
Dean 



Edit and mark up for 
printing 

Ed. 



Send to Media 
Production 



Ed. 



Edit and aark up for 
printing 

Ed. 



Send to Media 
Production 



Ed. 



— E 



rk up D4 for 
Inting 



Send D4 to Media 
Production 



Prepare special 
itens for Suiner 
School use 



Prepare special 
Iteatt for study 
centre display 





Prepare TV and 
radio coMwntarles 

CUA 
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ANNEX K 
(cont'dO 



eo) OP 
PHASE 3 



PRODUCTS 



Prepare "narking 
sclmie" end notes 
(If any) for tutor 
narked asslgnnents 
to send to corres- 
pondence tutois 

CIM 



Send to Kedia 
Production 



Correct proofs fron 
la Production 

EdVCT 




Marking schenes 



Coloured pages 



Final assessnent 



Notes ror race-to- 
face tutors 



Notes for regional 
tutors 



Glossaries or hand- 
books for students 



Correspondence 
narrative 



Correspondence 
raphlcs 



If -assessed 
asslgnnents 



Computer marked 
asslgnnents 



Tutor marked 
asslgnnents 



Special sunner 
school Items 



special stuoy 
centre Items 



repare students 
tudy calendar 



Yepare analysis of 
lues t1 on type, valid 
responses, points 
icored per response, 
for coiiMiter marked 
ttslnnm^nts Clik 



Send to Hedia 
Production 



Send to Data 
Processing 




bourse sunnairy to 
pro spectus 



"tik Set books 



Records/tapes 
for students 



^Reci 



omnended books 



Reader 



:xper1nenta1 kits 



students' study 
calendar 



Analysis for 
computer 




Radio progrannes 



idio conmentarles 



TV cownentarles 
TV programme 
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